ConcurrentRebalancersTest.TwoConcurrentRebalancers/1: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240501 20:10:56.616511  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 15793
I20240501 20:10:56.623219 15915 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:56.708178 16199 consensus_peers.cc:462] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection refused (error 111)
I20240501 20:10:56.836934 20692 raft_consensus.cc:1050] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:56.837751 20692 consensus_queue.cc:237] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:56.840493 18060 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:56.840667 17792 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:56.841041 16263 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:56.841240 21030 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:56.841768 21030 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:56.842257 21030 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:56.850184 21008 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.855564 15731 catalog_manager.cc:5484] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:56.858045 18058 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.860467 17792 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.864084 16263 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.888442  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 15793
W20240501 20:10:56.923987 17994 connection.cc:535] server connection from 127.8.68.193:37139 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:56.924448  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 15924
W20240501 20:10:56.924542 16200 connection.cc:535] client connection to 127.8.68.193:35513 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:56.925480 16200 consensus_peers.cc:485] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e -> Peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Couldn't send request to peer 51b81a5ae56b4dd99af1b0167c1da1e2. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:56.925961 16070 consensus_peers.cc:485] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Couldn't send request to peer 51b81a5ae56b4dd99af1b0167c1da1e2. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:56.930639 16043 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:56.933784 20159 raft_consensus.cc:1050] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
W20240501 20:10:56.934192 15719 catalog_manager.cc:4573] TS 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eef58fcca15041a39e685a7cdcb04f2b: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:56.934679 20159 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:56.936807 17792 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:56.937093 18058 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:56.937350 20820 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:56.937806 20820 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:10:56.938855 16199 consensus_peers.cc:485] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:56.941112 20820 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.941758 18060 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.945966 17792 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.946807 15734 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 10 to 11, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.953125 16263 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:56.954696 17792 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:10:56.954743 18060 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:10:56.955191 21147 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:10:56.955536 20820 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:10:56.957892 20159 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.958820 17792 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.958851 18058 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.961207 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c538581c71f646b5bc8d71cb6981f201 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:56.963340 15731 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 11 to 12, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:10:57.145408 16200 consensus_peers.cc:485] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:57.146870 16070 consensus_peers.cc:485] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Couldn't send request to peer 51b81a5ae56b4dd99af1b0167c1da1e2. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:57.206588 16199 consensus_peers.cc:462] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection refused (error 111)
I20240501 20:10:57.207547  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 15924
W20240501 20:10:57.242156 16071 connection.cc:535] client connection to 127.8.68.194:46677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:57.242602  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 16054
W20240501 20:10:57.243227 16071 consensus_peers.cc:485] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Couldn't send request to peer 754fcd3b82f041568f546c6205f44272. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:46677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:57.243356 16071 consensus_peers.cc:485] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Couldn't send request to peer 754fcd3b82f041568f546c6205f44272. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:46677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:57.243490 16071 consensus_peers.cc:485] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Couldn't send request to peer 754fcd3b82f041568f546c6205f44272. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:46677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:57.248692 16174 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:57.355835 16200 consensus_peers.cc:485] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:57.481343  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 16054
W20240501 20:10:57.514830 17993 connection.cc:535] server connection from 127.8.68.195:55659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:57.514835 17730 connection.cc:535] server connection from 127.8.68.195:36333 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:57.515321  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 16184
I20240501 20:10:57.521090 16305 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:57.736194 21157 raft_consensus.cc:482] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:57.736447 21157 raft_consensus.cc:504] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:57.738143 21157 leader_election.cc:288] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791)
W20240501 20:10:57.739710 17728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
W20240501 20:10:57.741528 17729 leader_election.cc:334] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Network error: Client connection negotiation failed: client connection to 127.8.68.194:46677: connect: Connection refused (error 111)
W20240501 20:10:57.741724 17728 leader_election.cc:334] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
I20240501 20:10:57.741869 17728 leader_election.cc:302] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9a0f0037700246bdb65e05aef95fb2be; no voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272
I20240501 20:10:57.742162 21157 raft_consensus.cc:2726] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:57.742607 17728 leader_election.cc:334] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:57.756251 20820 raft_consensus.cc:482] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:57.756446 20820 raft_consensus.cc:504] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:57.757164 20820 leader_election.cc:288] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791)
I20240501 20:10:57.757464  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 16184
W20240501 20:10:57.790810 17993 connection.cc:535] server connection from 127.8.68.196:43175 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:57.790858 17728 connection.cc:535] server connection from 127.8.68.196:56775 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:57.791457  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 17708
I20240501 20:10:57.838510 17834 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:57.969867 15719 catalog_manager.cc:4573] TS 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eef58fcca15041a39e685a7cdcb04f2b: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:57.975878 21165 raft_consensus.cc:482] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:57.976132 21165 raft_consensus.cc:504] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:57.977870 21165 leader_election.cc:288] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623)
W20240501 20:10:57.978811 17993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
W20240501 20:10:57.980043 17993 leader_election.cc:334] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
I20240501 20:10:58.025184 21157 raft_consensus.cc:482] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:58.025382 21157 raft_consensus.cc:504] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:58.025774 21170 raft_consensus.cc:482] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:58.026247  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 17708
I20240501 20:10:58.025964 21170 raft_consensus.cc:504] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:58.051324 21165 raft_consensus.cc:482] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:58.051573 21165 raft_consensus.cc:504] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:58.052675 21165 leader_election.cc:288] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791)
W20240501 20:10:58.054333 17994 leader_election.cc:334] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): Network error: Client connection negotiation failed: client connection to 127.8.68.195:46553: connect: Connection refused (error 111)
W20240501 20:10:58.054667 17993 leader_election.cc:334] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
I20240501 20:10:58.054806 17993 leader_election.cc:302] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cad00548f33849e7a19cecd760e1e429; no voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2
W20240501 20:10:58.055150 17993 leader_election.cc:334] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:58.055189 21165 raft_consensus.cc:2726] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:58.055977  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 17977
W20240501 20:10:58.056521 21168 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:57.978071 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.197:37623 (local address 127.8.68.199:45473)
0501 20:10:57.978656 (+   585us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:57.978673 (+    17us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:57.978781 (+   108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:58.026431 (+ 47650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:10:58.026437 (+     6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:10:58.026466 (+    29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:10:58.027126 (+   660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:10:58.027142 (+    16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:10:58.055769 (+ 28627us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.197:37623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":406,"threads_started":1}
W20240501 20:10:58.057752 17995 leader_election.cc:334] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Network error: Client connection negotiation failed: client connection to 127.8.68.197:37623: connect: Connection refused (error 111)
I20240501 20:10:58.057865 17995 leader_election.cc:302] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cad00548f33849e7a19cecd760e1e429; no voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:58.058180 21165 raft_consensus.cc:2726] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:58.061820 18099 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:58.255579 21165 raft_consensus.cc:482] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:58.255807 21165 raft_consensus.cc:504] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:58.256140  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 17977
I20240501 20:10:58.283211  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 15703
I20240501 20:10:58.284224 15764 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:58.525219  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 15703
2024-05-01T20:10:58Z chronyd exiting
I20240501 20:10:58.576020  8467 test_util.cc:175] -----------------------------------------------
I20240501 20:10:58.576102  8467 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0

Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN      ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 20:08:50.464221  8467 test_util.cc:268] Using random seed: 236228022
I20240501 20:08:50.474571  8467 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:50.474638  8467 ts_itest-base.cc:113] --------------
I20240501 20:08:50.474679  8467 ts_itest-base.cc:114] 6 tablet servers
I20240501 20:08:50.474736  8467 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:08:50.474776  8467 ts_itest-base.cc:116] --------------
2024-05-01T20:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:50Z Disabled control of system clock
I20240501 20:08:50.495272  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:39131
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:39131
--location_mapping_cmd=/tmp/dist-test-taskH_e3f3/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:08:50.783474  8479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:50.783730  8479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:50.783823  8479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:50.791005  8479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:50.791129  8479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:50.791203  8479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:50.791280  8479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:50.791339  8479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:50.799744  8479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskH_e3f3/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:39131
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:39131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:50.800806  8479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:50.802347  8479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:50.812669  8484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:50.813030  8485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:50.813602  8479 server_base.cc:1034] running on GCE node
W20240501 20:08:50.813966  8487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:50.814682  8479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:50.815879  8479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:50.817073  8479 hybrid_clock.cc:648] HybridClock initialized: now 1714594130817023 us; error 48 us; skew 500 ppm
I20240501 20:08:50.817361  8479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:50.819392  8479 webserver.cc:460] Webserver started at http://127.8.68.254:43443/ using document root <none> and password file <none>
I20240501 20:08:50.820044  8479 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:50.820145  8479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:50.820412  8479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:50.822777  8479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7f64f846da674e90b273568281328ac0"
format_stamp: "Formatted at 2024-05-01 20:08:50 on dist-test-slave-08p9"
I20240501 20:08:50.823432  8479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7f64f846da674e90b273568281328ac0"
format_stamp: "Formatted at 2024-05-01 20:08:50 on dist-test-slave-08p9"
I20240501 20:08:50.827884  8479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:08:50.830842  8493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:50.831961  8479 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:08:50.832109  8479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7f64f846da674e90b273568281328ac0"
format_stamp: "Formatted at 2024-05-01 20:08:50 on dist-test-slave-08p9"
I20240501 20:08:50.832274  8479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:50.848857  8479 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:08:50.849484  8479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:50.849704  8479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:50.864723  8479 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:39131
I20240501 20:08:50.864774  8544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:39131 every 8 connection(s)
I20240501 20:08:50.865877  8479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:50.869135  8545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:50.876569  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 8479
I20240501 20:08:50.876818  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:50.877525  8545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0: Bootstrap starting.
I20240501 20:08:50.880357  8545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:50.881340  8545 log.cc:824] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:50.884104  8545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0: No bootstrap required, opened a new log
I20240501 20:08:50.893343  8545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f64f846da674e90b273568281328ac0" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 39131 } }
I20240501 20:08:50.893744  8545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:50.893872  8545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f64f846da674e90b273568281328ac0, State: Initialized, Role: FOLLOWER
I20240501 20:08:50.894536  8545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f64f846da674e90b273568281328ac0" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 39131 } }
I20240501 20:08:50.894781  8545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:50.894902  8545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:50.895059  8545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:50.897310  8545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f64f846da674e90b273568281328ac0" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 39131 } }
I20240501 20:08:50.897852  8545 leader_election.cc:302] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f64f846da674e90b273568281328ac0; no voters: 
I20240501 20:08:50.898511  8545 leader_election.cc:288] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:08:50.898625  8550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:50.899372  8550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [term 1 LEADER]: Becoming Leader. State: Replica: 7f64f846da674e90b273568281328ac0, State: Running, Role: LEADER
I20240501 20:08:50.899914  8550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f64f846da674e90b273568281328ac0" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 39131 } }
I20240501 20:08:50.901074  8545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:50.904330  8552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f64f846da674e90b273568281328ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f64f846da674e90b273568281328ac0" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 39131 } } }
I20240501 20:08:50.904644  8552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:50.905156  8551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f64f846da674e90b273568281328ac0. Latest consensus state: current_term: 1 leader_uuid: "7f64f846da674e90b273568281328ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f64f846da674e90b273568281328ac0" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 39131 } } }
I20240501 20:08:50.905387  8551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:50.907446  8556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:50.913067  8556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:50.921331  8556 catalog_manager.cc:1347] Generated new cluster ID: c6642d2d8092459b814a20d9f4ce58b8
I20240501 20:08:50.921430  8556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:50.930682  8556 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:50.932181  8556 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:50.952273  8556 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7f64f846da674e90b273568281328ac0: Generated new TSK 0
I20240501 20:08:50.953070  8556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:50.962333  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--builtin_ntp_servers=127.8.68.212:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:08:51.263674  8570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:51.263931  8570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:51.264048  8570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:51.271178  8570 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:51.271278  8570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:51.271427  8570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240501 20:08:51.279896  8570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:51.281003  8570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:51.282619  8570 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:51.293380  8575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.293814  8576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.294145  8578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:51.294783  8570 server_base.cc:1034] running on GCE node
I20240501 20:08:51.295320  8570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:51.296123  8570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:51.297348  8570 hybrid_clock.cc:648] HybridClock initialized: now 1714594131297308 us; error 32 us; skew 500 ppm
I20240501 20:08:51.297638  8570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:51.299651  8570 webserver.cc:460] Webserver started at http://127.8.68.193:45879/ using document root <none> and password file <none>
I20240501 20:08:51.300303  8570 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:51.300410  8570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:51.300705  8570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:51.303211  8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-08p9"
I20240501 20:08:51.303923  8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-08p9"
I20240501 20:08:51.308504  8570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 20:08:51.311573  8584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:51.312752  8570 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:08:51.312894  8570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-08p9"
I20240501 20:08:51.313062  8570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:51.325816  8570 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:08:51.326555  8570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:51.326786  8570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:51.327602  8570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:51.328897  8570 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:51.328986  8570 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:51.329087  8570 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:51.329154  8570 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:51.358762  8570 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:45491
I20240501 20:08:51.358825  8696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:45491 every 8 connection(s)
I20240501 20:08:51.359988  8570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:51.365152  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 8570
I20240501 20:08:51.365353  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:51.368646  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--builtin_ntp_servers=127.8.68.212:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:51.373829  8697 heartbeater.cc:347] Connected to a master server at 127.8.68.254:39131
I20240501 20:08:51.374086  8697 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:51.374868  8697 heartbeater.cc:511] Master 127.8.68.254:39131 requested a full tablet report, sending...
I20240501 20:08:51.414093  8510 ts_manager.cc:194] Registered new tserver with Master: a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:08:51.415753  8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:41263
W20240501 20:08:51.676040  8700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:51.676298  8700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:51.676424  8700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:51.683553  8700 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:51.683669  8700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:51.683820  8700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240501 20:08:51.692108  8700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:51.693197  8700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:51.694794  8700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:51.705708  8707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.706068  8709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.706429  8706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:51.706887  8700 server_base.cc:1034] running on GCE node
I20240501 20:08:51.707357  8700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:51.708190  8700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:51.709370  8700 hybrid_clock.cc:648] HybridClock initialized: now 1714594131709323 us; error 41 us; skew 500 ppm
I20240501 20:08:51.709650  8700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:51.711617  8700 webserver.cc:460] Webserver started at http://127.8.68.194:35073/ using document root <none> and password file <none>
I20240501 20:08:51.712282  8700 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:51.712380  8700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:51.712671  8700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:51.715009  8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-08p9"
I20240501 20:08:51.715682  8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-08p9"
I20240501 20:08:51.720256  8700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.004s
I20240501 20:08:51.723336  8715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:51.724493  8700 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:08:51.724627  8700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-08p9"
I20240501 20:08:51.724807  8700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:51.741005  8700 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:08:51.741657  8700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:51.741896  8700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:51.742789  8700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:51.744052  8700 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:51.744146  8700 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:51.744246  8700 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:51.744314  8700 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:51.772295  8700 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:34923
I20240501 20:08:51.772365  8827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:34923 every 8 connection(s)
I20240501 20:08:51.773460  8700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:51.783155  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 8700
I20240501 20:08:51.783337  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:51.784652  8828 heartbeater.cc:347] Connected to a master server at 127.8.68.254:39131
I20240501 20:08:51.784900  8828 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:51.785753  8828 heartbeater.cc:511] Master 127.8.68.254:39131 requested a full tablet report, sending...
I20240501 20:08:51.786540  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--builtin_ntp_servers=127.8.68.212:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:51.839608  8510 ts_manager.cc:194] Registered new tserver with Master: 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923)
I20240501 20:08:51.840467  8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:56415
W20240501 20:08:52.107270  8831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:52.107522  8831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:52.107650  8831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:52.114667  8831 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:52.114782  8831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:52.114917  8831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240501 20:08:52.123283  8831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:52.124336  8831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:52.125870  8831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:52.136937  8837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.137301  8838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.137643  8840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:52.138116  8831 server_base.cc:1034] running on GCE node
I20240501 20:08:52.138623  8831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:52.139393  8831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:52.140579  8831 hybrid_clock.cc:648] HybridClock initialized: now 1714594132140532 us; error 44 us; skew 500 ppm
I20240501 20:08:52.140898  8831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:52.142832  8831 webserver.cc:460] Webserver started at http://127.8.68.195:44027/ using document root <none> and password file <none>
I20240501 20:08:52.143447  8831 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:52.143544  8831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:52.143831  8831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:52.146049  8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "02fb13c94e664ec280fad2968e515d19"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-08p9"
I20240501 20:08:52.146749  8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "02fb13c94e664ec280fad2968e515d19"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-08p9"
I20240501 20:08:52.151151  8831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:08:52.154021  8846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:52.155165  8831 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20240501 20:08:52.155301  8831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "02fb13c94e664ec280fad2968e515d19"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-08p9"
I20240501 20:08:52.155465  8831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:52.198457  8831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:08:52.199158  8831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:52.199400  8831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:52.200277  8831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:52.201669  8831 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:52.201768  8831 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:52.201866  8831 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:52.201942  8831 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:52.230870  8831 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:44647
I20240501 20:08:52.230943  8958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:44647 every 8 connection(s)
I20240501 20:08:52.232133  8831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:52.240960  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 8831
I20240501 20:08:52.241148  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:52.244233  8959 heartbeater.cc:347] Connected to a master server at 127.8.68.254:39131
I20240501 20:08:52.244477  8959 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:52.244711  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--builtin_ntp_servers=127.8.68.212:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:52.245218  8959 heartbeater.cc:511] Master 127.8.68.254:39131 requested a full tablet report, sending...
I20240501 20:08:52.286355  8510 ts_manager.cc:194] Registered new tserver with Master: 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647)
I20240501 20:08:52.287685  8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:51355
I20240501 20:08:52.418679  8697 heartbeater.cc:503] Master 127.8.68.254:39131 was elected leader, sending a full tablet report...
W20240501 20:08:52.545697  8962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:52.545930  8962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:52.546026  8962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:52.553016  8962 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:52.553112  8962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:52.553218  8962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:08:52.561533  8962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:52.562548  8962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:52.564034  8962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:52.574712  8968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.575196  8969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.575570  8971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:52.576170  8962 server_base.cc:1034] running on GCE node
I20240501 20:08:52.576766  8962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:52.577597  8962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:52.578805  8962 hybrid_clock.cc:648] HybridClock initialized: now 1714594132578753 us; error 37 us; skew 500 ppm
I20240501 20:08:52.579082  8962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:52.581017  8962 webserver.cc:460] Webserver started at http://127.8.68.196:41061/ using document root <none> and password file <none>
I20240501 20:08:52.581633  8962 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:52.581737  8962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:52.582016  8962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:52.584357  8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3b6c3b236a5d40bf99869152603b5524"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-08p9"
I20240501 20:08:52.585028  8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3b6c3b236a5d40bf99869152603b5524"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-08p9"
I20240501 20:08:52.589486  8962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20240501 20:08:52.592483  8977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:52.593796  8962 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:08:52.593930  8962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3b6c3b236a5d40bf99869152603b5524"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-08p9"
I20240501 20:08:52.594097  8962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:52.613322  8962 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:08:52.613968  8962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:52.614197  8962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:52.615100  8962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:52.616552  8962 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:52.616641  8962 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:52.616739  8962 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:52.616817  8962 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:52.643766  8962 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:38387
I20240501 20:08:52.643831  9089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:38387 every 8 connection(s)
I20240501 20:08:52.644857  8962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:52.649479  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 8962
I20240501 20:08:52.649628  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:52.652513  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--builtin_ntp_servers=127.8.68.212:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:52.656877  9090 heartbeater.cc:347] Connected to a master server at 127.8.68.254:39131
I20240501 20:08:52.657122  9090 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:52.657832  9090 heartbeater.cc:511] Master 127.8.68.254:39131 requested a full tablet report, sending...
I20240501 20:08:52.695139  8510 ts_manager.cc:194] Registered new tserver with Master: 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:08:52.696013  8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:57251
I20240501 20:08:52.843586  8828 heartbeater.cc:503] Master 127.8.68.254:39131 was elected leader, sending a full tablet report...
W20240501 20:08:52.953737  9093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:52.953979  9093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:52.954114  9093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:52.961131  9093 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:52.961236  9093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:52.961380  9093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:08:52.969478  9093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:52.970510  9093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:52.972044  9093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:52.982615  9099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.982901  9100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.983237  9102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:52.983860  9093 server_base.cc:1034] running on GCE node
I20240501 20:08:52.984334  9093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:52.985126  9093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:52.986340  9093 hybrid_clock.cc:648] HybridClock initialized: now 1714594132986293 us; error 40 us; skew 500 ppm
I20240501 20:08:52.986608  9093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:52.988551  9093 webserver.cc:460] Webserver started at http://127.8.68.197:44979/ using document root <none> and password file <none>
I20240501 20:08:52.989172  9093 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:52.989274  9093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:52.989562  9093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:52.991847  9093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4ea5761ef47546768883016105c52225"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-08p9"
I20240501 20:08:52.992532  9093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4ea5761ef47546768883016105c52225"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-08p9"
I20240501 20:08:52.996901  9093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20240501 20:08:52.999955  9108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:53.001099  9093 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:08:53.001245  9093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4ea5761ef47546768883016105c52225"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-08p9"
I20240501 20:08:53.001405  9093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:53.023322  9093 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:08:53.023939  9093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:53.024147  9093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:53.024914  9093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:53.026245  9093 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:53.026350  9093 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:53.026428  9093 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:53.026480  9093 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:53.052918  9093 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:45801
I20240501 20:08:53.052975  9220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:45801 every 8 connection(s)
I20240501 20:08:53.053977  9093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:53.056294  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 9093
I20240501 20:08:53.056486  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:53.059625  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:0
--local_ip_for_outbound_sockets=127.8.68.198
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--builtin_ntp_servers=127.8.68.212:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:53.066030  9221 heartbeater.cc:347] Connected to a master server at 127.8.68.254:39131
I20240501 20:08:53.066329  9221 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:53.067137  9221 heartbeater.cc:511] Master 127.8.68.254:39131 requested a full tablet report, sending...
I20240501 20:08:53.106144  8510 ts_manager.cc:194] Registered new tserver with Master: 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:08:53.107087  8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:51369
I20240501 20:08:53.290120  8959 heartbeater.cc:503] Master 127.8.68.254:39131 was elected leader, sending a full tablet report...
W20240501 20:08:53.366899  9224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:53.367163  9224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:53.367305  9224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:53.374349  9224 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:53.374462  9224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:53.374608  9224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240501 20:08:53.382783  9224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:39131
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:53.383875  9224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:53.385454  9224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:53.396142  9230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:53.396574  9231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:53.396975  9233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:53.397495  9224 server_base.cc:1034] running on GCE node
I20240501 20:08:53.398023  9224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:53.398898  9224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:53.400084  9224 hybrid_clock.cc:648] HybridClock initialized: now 1714594133400043 us; error 32 us; skew 500 ppm
I20240501 20:08:53.400346  9224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:53.402315  9224 webserver.cc:460] Webserver started at http://127.8.68.198:44897/ using document root <none> and password file <none>
I20240501 20:08:53.402973  9224 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:53.403064  9224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:53.403323  9224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:53.405651  9224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "23543295bda14131a70bd31c573454f6"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-08p9"
I20240501 20:08:53.406396  9224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "23543295bda14131a70bd31c573454f6"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-08p9"
I20240501 20:08:53.410893  9224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240501 20:08:53.413807  9239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:53.414942  9224 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:08:53.415077  9224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "23543295bda14131a70bd31c573454f6"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-08p9"
I20240501 20:08:53.415241  9224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:53.431457  9224 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:08:53.432119  9224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:53.432401  9224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:53.433480  9224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:53.435174  9224 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:53.435287  9224 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:53.435387  9224 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:53.435460  9224 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:08:53.464033  9224 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:39123
I20240501 20:08:53.464088  9351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:39123 every 8 connection(s)
I20240501 20:08:53.465210  9224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:08:53.474089  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 9224
I20240501 20:08:53.474244  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:08:53.476872  9352 heartbeater.cc:347] Connected to a master server at 127.8.68.254:39131
I20240501 20:08:53.477115  9352 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:53.477813  9352 heartbeater.cc:511] Master 127.8.68.254:39131 requested a full tablet report, sending...
I20240501 20:08:53.516984  8510 ts_manager.cc:194] Registered new tserver with Master: 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:08:53.517865  8510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:59105
I20240501 20:08:53.526890  8467 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 20:08:53.544849  8467 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 20:08:53.698588  9090 heartbeater.cc:503] Master 127.8.68.254:39131 was elected leader, sending a full tablet report...
I20240501 20:08:54.109611  9221 heartbeater.cc:503] Master 127.8.68.254:39131 was elected leader, sending a full tablet report...
I20240501 20:08:54.520115  9352 heartbeater.cc:503] Master 127.8.68.254:39131 was elected leader, sending a full tablet report...
I20240501 20:08:57.298799  8510 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:08:57.348491  9155 tablet_service.cc:1467] Processing CreateTablet for tablet 514b5a047b694b878818b43f2e8c2f80 (DEFAULT_TABLE table=rebalance_test_table_0 [id=51cbb47e48124e5c8215d4a89d8c5db1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:57.350502  9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514b5a047b694b878818b43f2e8c2f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:57.351389  9156 tablet_service.cc:1467] Processing CreateTablet for tablet 14969b6bae9c44a19f774abc330216ec (DEFAULT_TABLE table=rebalance_test_table_0 [id=51cbb47e48124e5c8215d4a89d8c5db1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:57.352557  9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14969b6bae9c44a19f774abc330216ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:57.358062  9287 tablet_service.cc:1467] Processing CreateTablet for tablet b88d93c63032451eb0e6904aaac0001b (DEFAULT_TABLE table=rebalance_test_table_0 [id=51cbb47e48124e5c8215d4a89d8c5db1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:57.358562  8763 tablet_service.cc:1467] Processing CreateTablet for tablet 14969b6bae9c44a19f774abc330216ec (DEFAULT_TABLE table=rebalance_test_table_0 [id=51cbb47e48124e5c8215d4a89d8c5db1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:57.359439  9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88d93c63032451eb0e6904aaac0001b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:57.359947  8763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14969b6bae9c44a19f774abc330216ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:57.366461  8893 tablet_service.cc:1467] Processing CreateTablet for tablet 514b5a047b694b878818b43f2e8c2f80 (DEFAULT_TABLE table=rebalance_test_table_0 [id=51cbb47e48124e5c8215d4a89d8c5db1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:57.368461  8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514b5a047b694b878818b43f2e8c2f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:57.367519  8894 tablet_service.cc:1467] Processing CreateTablet for tablet b88d93c63032451eb0e6904aaac0001b (DEFAULT_TABLE table=rebalance_test_table_0 [id=51cbb47e48124e5c8215d4a89d8c5db1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:57.369352  8894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88d93c63032451eb0e6904aaac0001b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:57.374308  9378 tablet_bootstrap.cc:492] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:08:57.377158  9378 tablet_bootstrap.cc:654] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:57.378082  9378 log.cc:824] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:57.382196  9378 tablet_bootstrap.cc:492] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:08:57.382645  9378 ts_tablet_manager.cc:1367] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240501 20:08:57.390736  9378 raft_consensus.cc:348] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:57.391352  9378 raft_consensus.cc:374] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:57.391521  9378 raft_consensus.cc:729] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:08:57.392175  9378 consensus_queue.cc:260] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:57.393498  9380 tablet_bootstrap.cc:492] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:08:57.393019  8631 tablet_service.cc:1467] Processing CreateTablet for tablet 514b5a047b694b878818b43f2e8c2f80 (DEFAULT_TABLE table=rebalance_test_table_0 [id=51cbb47e48124e5c8215d4a89d8c5db1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:57.396780  8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514b5a047b694b878818b43f2e8c2f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:57.397078  8632 tablet_service.cc:1467] Processing CreateTablet for tablet b88d93c63032451eb0e6904aaac0001b (DEFAULT_TABLE table=rebalance_test_table_0 [id=51cbb47e48124e5c8215d4a89d8c5db1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:57.398418  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88d93c63032451eb0e6904aaac0001b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:57.399288  9380 tablet_bootstrap.cc:654] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:57.399683  9377 tablet_bootstrap.cc:492] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:08:57.400352  9380 log.cc:824] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:57.402383  9378 ts_tablet_manager.cc:1398] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.020s	user 0.014s	sys 0.004s
I20240501 20:08:57.404042  9377 tablet_bootstrap.cc:654] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:57.405149  9377 log.cc:824] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:57.415817  9025 tablet_service.cc:1467] Processing CreateTablet for tablet 14969b6bae9c44a19f774abc330216ec (DEFAULT_TABLE table=rebalance_test_table_0 [id=51cbb47e48124e5c8215d4a89d8c5db1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:57.417364  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14969b6bae9c44a19f774abc330216ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:57.418416  9382 tablet_bootstrap.cc:492] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:08:57.419162  9377 tablet_bootstrap.cc:492] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:08:57.419422  9377 ts_tablet_manager.cc:1367] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.020s	user 0.005s	sys 0.005s
I20240501 20:08:57.420840  9385 tablet_bootstrap.cc:492] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:08:57.422447  9382 tablet_bootstrap.cc:654] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:57.423583  9382 log.cc:824] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:57.424319  9385 tablet_bootstrap.cc:654] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:57.425419  9385 log.cc:824] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:57.430830  9380 tablet_bootstrap.cc:492] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:08:57.430866  9385 tablet_bootstrap.cc:492] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:08:57.431082  9385 ts_tablet_manager.cc:1367] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.001s
I20240501 20:08:57.431105  9380 ts_tablet_manager.cc:1367] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.005s
I20240501 20:08:57.430935  9377 raft_consensus.cc:348] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:57.431387  9377 raft_consensus.cc:374] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:57.431509  9377 raft_consensus.cc:729] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:08:57.432302  9377 consensus_queue.cc:260] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:57.439731  9380 raft_consensus.cc:348] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:57.440238  9380 raft_consensus.cc:374] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:57.440373  9380 raft_consensus.cc:729] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:08:57.440948  9385 raft_consensus.cc:348] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:57.441093  9380 consensus_queue.cc:260] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:57.441346  9385 raft_consensus.cc:374] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:57.441444  9385 raft_consensus.cc:729] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:08:57.442359  9382 tablet_bootstrap.cc:492] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:08:57.442636  9382 ts_tablet_manager.cc:1367] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.024s	user 0.004s	sys 0.012s
I20240501 20:08:57.443806  9389 tablet_bootstrap.cc:492] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:08:57.444989  9385 consensus_queue.cc:260] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:57.449090  9377 ts_tablet_manager.cc:1398] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.029s	user 0.023s	sys 0.000s
I20240501 20:08:57.449592  9377 tablet_bootstrap.cc:492] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:08:57.449738  9389 tablet_bootstrap.cc:654] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:57.451113  9377 tablet_bootstrap.cc:654] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:57.451366  9380 ts_tablet_manager.cc:1398] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.020s	user 0.014s	sys 0.000s
I20240501 20:08:57.455273  9380 tablet_bootstrap.cc:492] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:08:57.455956  9382 raft_consensus.cc:348] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:57.456416  9382 raft_consensus.cc:374] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:57.456553  9382 raft_consensus.cc:729] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:08:57.456710  9380 tablet_bootstrap.cc:654] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:57.457181  9382 consensus_queue.cc:260] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:57.458675  9385 ts_tablet_manager.cc:1398] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.014s
I20240501 20:08:57.459110  9385 tablet_bootstrap.cc:492] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:08:57.460577  9385 tablet_bootstrap.cc:654] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:57.460902  9377 tablet_bootstrap.cc:492] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:08:57.460970  9382 ts_tablet_manager.cc:1398] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.006s
I20240501 20:08:57.461088  9377 ts_tablet_manager.cc:1367] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.007s
I20240501 20:08:57.461995  9377 raft_consensus.cc:348] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:08:57.462229  9377 raft_consensus.cc:374] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:57.462704  9389 log.cc:824] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:57.462869  9377 raft_consensus.cc:729] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:08:57.463016  9380 tablet_bootstrap.cc:492] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:08:57.463138  9380 ts_tablet_manager.cc:1367] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240501 20:08:57.463303  9377 consensus_queue.cc:260] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:08:57.463924  9377 ts_tablet_manager.cc:1398] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:08:57.464749  9380 raft_consensus.cc:348] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:08:57.464936  9380 raft_consensus.cc:374] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:57.465034  9380 raft_consensus.cc:729] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:08:57.465242  9380 consensus_queue.cc:260] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:08:57.465718  9380 ts_tablet_manager.cc:1398] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:08:57.466219  9385 tablet_bootstrap.cc:492] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:08:57.466527  9385 ts_tablet_manager.cc:1367] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240501 20:08:57.467881  9385 raft_consensus.cc:348] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:08:57.468137  9385 raft_consensus.cc:374] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:57.468286  9385 raft_consensus.cc:729] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
W20240501 20:08:57.469033  9353 tablet.cc:2369] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:57.468607  9385 consensus_queue.cc:260] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:08:57.470271  9385 ts_tablet_manager.cc:1398] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:08:57.474211  9389 tablet_bootstrap.cc:492] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:08:57.474558  9389 ts_tablet_manager.cc:1367] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.031s	user 0.006s	sys 0.007s
I20240501 20:08:57.479987  9389 raft_consensus.cc:348] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:57.480278  9389 raft_consensus.cc:374] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:57.480365  9389 raft_consensus.cc:729] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:08:57.480854  9389 consensus_queue.cc:260] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:57.482131  9389 ts_tablet_manager.cc:1398] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.002s
W20240501 20:08:57.486649  8960 tablet.cc:2369] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:08:57.528451  8829 tablet.cc:2369] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:57.533629  9388 raft_consensus.cc:482] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:57.533852  9388 raft_consensus.cc:504] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:08:57.535466  9388 leader_election.cc:288] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647)
I20240501 20:08:57.542217  8914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514b5a047b694b878818b43f2e8c2f80" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:08:57.542320  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514b5a047b694b878818b43f2e8c2f80" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:08:57.542665  8914 raft_consensus.cc:2454] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 0.
I20240501 20:08:57.542737  8652 raft_consensus.cc:2454] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 0.
I20240501 20:08:57.543186  9109 leader_election.cc:302] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:08:57.543537  9388 raft_consensus.cc:2781] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:57.543648  9388 raft_consensus.cc:482] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:57.543730  9388 raft_consensus.cc:3037] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:57.545616  9388 raft_consensus.cc:504] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:08:57.546109  9388 leader_election.cc:288] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 election: Requested vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647)
I20240501 20:08:57.546473  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514b5a047b694b878818b43f2e8c2f80" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:08:57.546511  8914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514b5a047b694b878818b43f2e8c2f80" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19"
I20240501 20:08:57.546622  8652 raft_consensus.cc:3037] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:57.546641  8914 raft_consensus.cc:3037] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:57.548480  8652 raft_consensus.cc:2454] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 1.
I20240501 20:08:57.548540  8914 raft_consensus.cc:2454] T 514b5a047b694b878818b43f2e8c2f80 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 1.
I20240501 20:08:57.548837  9110 leader_election.cc:302] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea5761ef47546768883016105c52225, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:08:57.549088  9388 raft_consensus.cc:2781] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:57.549460  9388 raft_consensus.cc:686] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea5761ef47546768883016105c52225, State: Running, Role: LEADER
I20240501 20:08:57.549836  9388 consensus_queue.cc:237] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:08:57.552587  9394 raft_consensus.cc:482] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:57.552806  9394 raft_consensus.cc:504] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:57.554353  9394 leader_election.cc:288] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:08:57.554237  8510 catalog_manager.cc:5484] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ea5761ef47546768883016105c52225 (127.8.68.197). New cstate: current_term: 1 leader_uuid: "4ea5761ef47546768883016105c52225" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
W20240501 20:08:57.557734  9222 tablet.cc:2369] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:57.562721  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14969b6bae9c44a19f774abc330216ec" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:08:57.563002  9176 raft_consensus.cc:2454] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:08:57.563102  9045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14969b6bae9c44a19f774abc330216ec" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:08:57.563464  8718 leader_election.cc:302] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4e66c1f8df41d3a228b1f8b8ece5d8, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:08:57.563534  9045 raft_consensus.cc:2454] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:08:57.563755  9394 raft_consensus.cc:2781] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:57.563897  9394 raft_consensus.cc:482] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:57.563987  9394 raft_consensus.cc:3037] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:57.565981  9394 raft_consensus.cc:504] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:57.566504  9394 leader_election.cc:288] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:08:57.566859  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14969b6bae9c44a19f774abc330216ec" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:08:57.566926  9045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14969b6bae9c44a19f774abc330216ec" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:08:57.566998  9176 raft_consensus.cc:3037] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:57.567091  9045 raft_consensus.cc:3037] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:57.568945  9045 raft_consensus.cc:2454] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:08:57.568979  9176 raft_consensus.cc:2454] T 14969b6bae9c44a19f774abc330216ec P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:08:57.569725  8717 leader_election.cc:302] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:08:57.569990  9394 raft_consensus.cc:2781] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:57.570397  9394 raft_consensus.cc:686] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:08:57.570884  9394 consensus_queue.cc:237] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:57.574504  8510 catalog_manager.cc:5484] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } }
W20240501 20:08:57.615137  8698 tablet.cc:2369] T 514b5a047b694b878818b43f2e8c2f80 P a0e6c99bc66b433f9d5cf9340dad0cc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:08:57.649475  9091 tablet.cc:2369] T 14969b6bae9c44a19f774abc330216ec P 3b6c3b236a5d40bf99869152603b5524: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:57.744650  9381 raft_consensus.cc:482] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:57.744885  9381 raft_consensus.cc:504] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:57.746510  9381 leader_election.cc:288] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:08:57.754546  8914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d93c63032451eb0e6904aaac0001b" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:08:57.754611  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d93c63032451eb0e6904aaac0001b" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:08:57.754758  8914 raft_consensus.cc:2454] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:08:57.754827  8652 raft_consensus.cc:2454] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:08:57.755208  9240 leader_election.cc:302] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 23543295bda14131a70bd31c573454f6; no voters: 
I20240501 20:08:57.755537  9381 raft_consensus.cc:2781] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:57.755671  9381 raft_consensus.cc:482] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:57.755841  9381 raft_consensus.cc:3037] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:57.758066  9381 raft_consensus.cc:504] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:57.758626  9381 leader_election.cc:288] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Requested vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:08:57.758915  8914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d93c63032451eb0e6904aaac0001b" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19"
I20240501 20:08:57.759042  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d93c63032451eb0e6904aaac0001b" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:08:57.759105  8914 raft_consensus.cc:3037] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:57.759263  8652 raft_consensus.cc:3037] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:57.760848  8914 raft_consensus.cc:2454] T b88d93c63032451eb0e6904aaac0001b P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:08:57.761144  9240 leader_election.cc:302] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 23543295bda14131a70bd31c573454f6; no voters: 
I20240501 20:08:57.761176  8652 raft_consensus.cc:2454] T b88d93c63032451eb0e6904aaac0001b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:08:57.761369  9381 raft_consensus.cc:2781] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:57.761787  9381 raft_consensus.cc:686] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [term 1 LEADER]: Becoming Leader. State: Replica: 23543295bda14131a70bd31c573454f6, State: Running, Role: LEADER
I20240501 20:08:57.762198  9381 consensus_queue.cc:237] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:57.766364  8510 catalog_manager.cc:5484] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23543295bda14131a70bd31c573454f6 (127.8.68.198). New cstate: current_term: 1 leader_uuid: "23543295bda14131a70bd31c573454f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:58.074081  9411 consensus_queue.cc:1035] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:58.082554  9411 consensus_queue.cc:1035] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:58.092363  9419 consensus_queue.cc:1035] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:58.107123  9419 consensus_queue.cc:1035] T 14969b6bae9c44a19f774abc330216ec P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:58.121328  9410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:58.121593  9410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:58.128691  9410 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 20:08:58.229496  9381 consensus_queue.cc:1035] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:58.279175  9381 consensus_queue.cc:1035] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 289.756 ms
  time per row: 0.144878 ms
I20240501 20:08:58.554339  8509 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:08:58.562520  8763 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0f18068e024139a123f055c29b36b5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9d8b6a1a3dbe4c6d986f31a47c548383]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.562837  9025 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0f18068e024139a123f055c29b36b5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9d8b6a1a3dbe4c6d986f31a47c548383]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.562991  8763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f0f18068e024139a123f055c29b36b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.563251  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f0f18068e024139a123f055c29b36b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.563987  8632 tablet_service.cc:1467] Processing CreateTablet for tablet e93c8da097b64b6ead9bbb1168fd35e7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9d8b6a1a3dbe4c6d986f31a47c548383]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.564175  9287 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0f18068e024139a123f055c29b36b5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9d8b6a1a3dbe4c6d986f31a47c548383]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.564402  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93c8da097b64b6ead9bbb1168fd35e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.564553  9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f0f18068e024139a123f055c29b36b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.569836  9024 tablet_service.cc:1467] Processing CreateTablet for tablet ed26eb2563fd44748ff4b080e68c922d (DEFAULT_TABLE table=rebalance_test_table_1 [id=9d8b6a1a3dbe4c6d986f31a47c548383]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.570333  9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed26eb2563fd44748ff4b080e68c922d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.571169  8893 tablet_service.cc:1467] Processing CreateTablet for tablet e93c8da097b64b6ead9bbb1168fd35e7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9d8b6a1a3dbe4c6d986f31a47c548383]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.571615  8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93c8da097b64b6ead9bbb1168fd35e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.573711  9486 tablet_bootstrap.cc:492] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:08:58.574908  8762 tablet_service.cc:1467] Processing CreateTablet for tablet ed26eb2563fd44748ff4b080e68c922d (DEFAULT_TABLE table=rebalance_test_table_1 [id=9d8b6a1a3dbe4c6d986f31a47c548383]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.575371  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed26eb2563fd44748ff4b080e68c922d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.575395  9486 tablet_bootstrap.cc:654] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.579069  9286 tablet_service.cc:1467] Processing CreateTablet for tablet ed26eb2563fd44748ff4b080e68c922d (DEFAULT_TABLE table=rebalance_test_table_1 [id=9d8b6a1a3dbe4c6d986f31a47c548383]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.579537  9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed26eb2563fd44748ff4b080e68c922d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.580646  9487 tablet_bootstrap.cc:492] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:08:58.581166  9485 tablet_bootstrap.cc:492] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:08:58.582334  9487 tablet_bootstrap.cc:654] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.583292  9489 tablet_bootstrap.cc:492] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:08:58.583242  9156 tablet_service.cc:1467] Processing CreateTablet for tablet e93c8da097b64b6ead9bbb1168fd35e7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9d8b6a1a3dbe4c6d986f31a47c548383]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.583653  9485 tablet_bootstrap.cc:654] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.583657  9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93c8da097b64b6ead9bbb1168fd35e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.584959  9489 tablet_bootstrap.cc:654] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.589097  9489 tablet_bootstrap.cc:492] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:08:58.590258  9489 ts_tablet_manager.cc:1367] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240501 20:08:58.591400  9489 raft_consensus.cc:348] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:08:58.591814  9485 tablet_bootstrap.cc:492] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:08:58.592039  9485 ts_tablet_manager.cc:1367] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.003s
I20240501 20:08:58.593050  9485 raft_consensus.cc:348] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:58.593500  9489 raft_consensus.cc:374] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.593606  9489 raft_consensus.cc:729] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.593931  9485 raft_consensus.cc:374] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.594099  9485 raft_consensus.cc:729] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.594398  9485 consensus_queue.cc:260] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:58.594432  9489 consensus_queue.cc:260] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:08:58.595151  9485 ts_tablet_manager.cc:1398] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:08:58.595445  9485 tablet_bootstrap.cc:492] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:08:58.596930  9485 tablet_bootstrap.cc:654] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.598320  9493 tablet_bootstrap.cc:492] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:08:58.601234  9493 tablet_bootstrap.cc:654] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.601464  9487 tablet_bootstrap.cc:492] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:08:58.601858  9489 ts_tablet_manager.cc:1398] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.011s	user 0.000s	sys 0.002s
I20240501 20:08:58.602730  9487 ts_tablet_manager.cc:1367] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.022s	user 0.001s	sys 0.004s
I20240501 20:08:58.602943  9484 tablet_bootstrap.cc:492] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:08:58.603566  9487 raft_consensus.cc:348] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:08:58.603884  9487 raft_consensus.cc:374] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.604703  9486 tablet_bootstrap.cc:492] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:08:58.604831  9487 raft_consensus.cc:729] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.604904  9486 ts_tablet_manager.cc:1367] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.031s	user 0.000s	sys 0.006s
I20240501 20:08:58.605235  9485 tablet_bootstrap.cc:492] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:08:58.605132  9487 consensus_queue.cc:260] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:08:58.605445  9485 ts_tablet_manager.cc:1367] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.007s
I20240501 20:08:58.606154  9486 raft_consensus.cc:348] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:58.606379  9487 ts_tablet_manager.cc:1398] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:08:58.606508  9486 raft_consensus.cc:374] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.606458  9485 raft_consensus.cc:348] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:58.606644  9485 raft_consensus.cc:374] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.606722  9485 raft_consensus.cc:729] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.606685  9486 raft_consensus.cc:729] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.606952  9485 consensus_queue.cc:260] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:58.607415  9485 ts_tablet_manager.cc:1398] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:08:58.607784  9486 consensus_queue.cc:260] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:58.608146  9493 tablet_bootstrap.cc:492] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:08:58.608276  9493 ts_tablet_manager.cc:1367] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.004s
I20240501 20:08:58.608994  9493 raft_consensus.cc:348] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:08:58.609141  9493 raft_consensus.cc:374] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.609225  9493 raft_consensus.cc:729] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.609563  9493 consensus_queue.cc:260] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:08:58.610110  9486 ts_tablet_manager.cc:1398] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 20:08:58.610152  9493 ts_tablet_manager.cc:1398] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:08:58.610368  9486 tablet_bootstrap.cc:492] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:08:58.611061  9484 tablet_bootstrap.cc:654] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.612223  9486 tablet_bootstrap.cc:654] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.613720  9486 tablet_bootstrap.cc:492] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:08:58.613857  9486 ts_tablet_manager.cc:1367] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:08:58.614025  9484 tablet_bootstrap.cc:492] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:08:58.614140  9484 ts_tablet_manager.cc:1367] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240501 20:08:58.614425  9486 raft_consensus.cc:348] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:58.614581  9486 raft_consensus.cc:374] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.614643  9486 raft_consensus.cc:729] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.614858  9486 consensus_queue.cc:260] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:58.614996  9484 raft_consensus.cc:348] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:58.615162  9484 raft_consensus.cc:374] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.615247  9486 ts_tablet_manager.cc:1398] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240501 20:08:58.615252  9484 raft_consensus.cc:729] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.615461  9484 consensus_queue.cc:260] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:58.616083  9484 ts_tablet_manager.cc:1398] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:08:58.616377  9484 tablet_bootstrap.cc:492] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:08:58.617764  9484 tablet_bootstrap.cc:654] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.619375  9484 tablet_bootstrap.cc:492] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:08:58.619513  9484 ts_tablet_manager.cc:1367] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:08:58.620028  9484 raft_consensus.cc:348] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:58.620182  9484 raft_consensus.cc:374] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.620245  9484 raft_consensus.cc:729] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.620518  9484 consensus_queue.cc:260] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:58.620965  9484 ts_tablet_manager.cc:1398] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240501 20:08:58.690412  9412 raft_consensus.cc:482] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:58.690611  9412 raft_consensus.cc:504] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:08:58.692059  9412 leader_election.cc:288] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:08:58.700201  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93c8da097b64b6ead9bbb1168fd35e7" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:08:58.700186  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93c8da097b64b6ead9bbb1168fd35e7" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:08:58.700423  8652 raft_consensus.cc:2454] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:08:58.700438  9176 raft_consensus.cc:2454] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:08:58.700918  8849 leader_election.cc:302] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:08:58.701282  9412 raft_consensus.cc:2781] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:58.701395  9412 raft_consensus.cc:482] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:58.701485  9412 raft_consensus.cc:3037] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.703608  9412 raft_consensus.cc:504] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:08:58.704093  9412 leader_election.cc:288] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Requested vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:08:58.704466  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93c8da097b64b6ead9bbb1168fd35e7" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:08:58.704471  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93c8da097b64b6ead9bbb1168fd35e7" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:08:58.704625  8652 raft_consensus.cc:3037] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.704639  9176 raft_consensus.cc:3037] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.706421  8652 raft_consensus.cc:2454] T e93c8da097b64b6ead9bbb1168fd35e7 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:08:58.706638  9176 raft_consensus.cc:2454] T e93c8da097b64b6ead9bbb1168fd35e7 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:08:58.706753  8848 leader_election.cc:302] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:08:58.707002  9412 raft_consensus.cc:2781] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:58.707360  9412 raft_consensus.cc:686] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Running, Role: LEADER
I20240501 20:08:58.707811  9412 consensus_queue.cc:237] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:08:58.708807  9419 raft_consensus.cc:482] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:58.708950  9419 raft_consensus.cc:504] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:58.709887  9419 leader_election.cc:288] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:08:58.710711  9045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0f18068e024139a123f055c29b36b5" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:08:58.710951  9045 raft_consensus.cc:2454] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:08:58.710892  8508 catalog_manager.cc:5484] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fb13c94e664ec280fad2968e515d19 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "02fb13c94e664ec280fad2968e515d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:58.711362  8717 leader_election.cc:302] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:08:58.711592  9419 raft_consensus.cc:2781] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:58.711715  9419 raft_consensus.cc:482] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:58.711792  9419 raft_consensus.cc:3037] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.714015  9419 raft_consensus.cc:504] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:58.714697  9419 leader_election.cc:288] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:08:58.715065  9045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0f18068e024139a123f055c29b36b5" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:08:58.715237  9045 raft_consensus.cc:3037] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.717044  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0f18068e024139a123f055c29b36b5" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:08:58.717209  9045 raft_consensus.cc:2454] T 3f0f18068e024139a123f055c29b36b5 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:08:58.717069  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0f18068e024139a123f055c29b36b5" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:08:58.717466  9307 raft_consensus.cc:2454] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:08:58.717864  8717 leader_election.cc:302] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:08:58.718108  9419 raft_consensus.cc:2781] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:58.718232  9419 raft_consensus.cc:686] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:08:58.718492  9419 consensus_queue.cc:237] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:08:58.720638  8508 catalog_manager.cc:5484] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:58.937438  9381 raft_consensus.cc:482] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:58.937601  9381 raft_consensus.cc:504] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:58.938344  9381 leader_election.cc:288] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:08:58.955473  8783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed26eb2563fd44748ff4b080e68c922d" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" is_pre_election: true
I20240501 20:08:58.955672  9045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed26eb2563fd44748ff4b080e68c922d" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:08:58.955765  8783 raft_consensus.cc:2454] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:08:58.955870  9045 raft_consensus.cc:2454] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:08:58.956176  9240 leader_election.cc:302] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:08:58.956477  9381 raft_consensus.cc:2781] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:58.956570  9381 raft_consensus.cc:482] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:58.956662  9381 raft_consensus.cc:3037] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.958395  9381 raft_consensus.cc:504] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:58.959046  9381 leader_election.cc:288] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:08:58.959388  8783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed26eb2563fd44748ff4b080e68c922d" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
I20240501 20:08:58.959503  9045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed26eb2563fd44748ff4b080e68c922d" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:08:58.959559  8783 raft_consensus.cc:3037] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.959671  9045 raft_consensus.cc:3037] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.961385  9045 raft_consensus.cc:2454] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:08:58.961719  9241 leader_election.cc:302] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3b6c3b236a5d40bf99869152603b5524; no voters: 
I20240501 20:08:58.961884  8783 raft_consensus.cc:2454] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:08:58.961946  9381 raft_consensus.cc:2781] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:58.962323  9381 raft_consensus.cc:686] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [term 1 LEADER]: Becoming Leader. State: Replica: 23543295bda14131a70bd31c573454f6, State: Running, Role: LEADER
I20240501 20:08:58.962595  9381 consensus_queue.cc:237] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:08:58.965304  8508 catalog_manager.cc:5484] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23543295bda14131a70bd31c573454f6 (127.8.68.198). New cstate: current_term: 1 leader_uuid: "23543295bda14131a70bd31c573454f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:59.192152  9306 raft_consensus.cc:3037] T 3f0f18068e024139a123f055c29b36b5 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:59.194958  9419 consensus_queue.cc:1035] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:59.205024  9425 consensus_queue.cc:1035] T 3f0f18068e024139a123f055c29b36b5 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:59.266381  9520 consensus_queue.cc:1035] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:59.273309  9522 consensus_queue.cc:1035] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:59.344847  9513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:59.345116  9513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:59.352566  9513 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:08:59.487304  9045 raft_consensus.cc:1261] T ed26eb2563fd44748ff4b080e68c922d P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Refusing update from remote peer 23543295bda14131a70bd31c573454f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:59.495179  9381 consensus_queue.cc:1035] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:59.498682  8783 raft_consensus.cc:1261] T ed26eb2563fd44748ff4b080e68c922d P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Refusing update from remote peer 23543295bda14131a70bd31c573454f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:59.499375  9381 consensus_queue.cc:1035] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:59.558966  9516 mvcc.cc:205] Tried to move back new op lower bound from 7022977595325566976 to 7022977593192902656. Current Snapshot: MvccSnapshot[applied={T|T < 7022977595325566976}]
I20240501 20:08:59.574086  9515 mvcc.cc:205] Tried to move back new op lower bound from 7022977595325566976 to 7022977593192902656. Current Snapshot: MvccSnapshot[applied={T|T < 7022977595325566976}]

INSERT report
    rows total: 2000
    time total: 228.446 ms
  time per row: 0.114223 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:09:00.583475  8508 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:00.592787  8893 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4e4fa3e651453ba0344d5dc2607700 (DEFAULT_TABLE table=rebalance_test_table_2 [id=be7f1e6a510b4bce8e966c8515685cff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.593056  9156 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4e4fa3e651453ba0344d5dc2607700 (DEFAULT_TABLE table=rebalance_test_table_2 [id=be7f1e6a510b4bce8e966c8515685cff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.593245  8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4e4fa3e651453ba0344d5dc2607700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.593418  9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4e4fa3e651453ba0344d5dc2607700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.593459  8632 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4e4fa3e651453ba0344d5dc2607700 (DEFAULT_TABLE table=rebalance_test_table_2 [id=be7f1e6a510b4bce8e966c8515685cff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.593916  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4e4fa3e651453ba0344d5dc2607700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.596974  9287 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc72717208547689b45b9c789c8f3f9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=be7f1e6a510b4bce8e966c8515685cff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.597105  8631 tablet_service.cc:1467] Processing CreateTablet for tablet 26ad523451b64159877795981558e17c (DEFAULT_TABLE table=rebalance_test_table_2 [id=be7f1e6a510b4bce8e966c8515685cff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.597404  9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc72717208547689b45b9c789c8f3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.597486  8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ad523451b64159877795981558e17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.598644  8762 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc72717208547689b45b9c789c8f3f9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=be7f1e6a510b4bce8e966c8515685cff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.599017  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc72717208547689b45b9c789c8f3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.602082  9025 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc72717208547689b45b9c789c8f3f9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=be7f1e6a510b4bce8e966c8515685cff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.602543  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc72717208547689b45b9c789c8f3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.606122  8894 tablet_service.cc:1467] Processing CreateTablet for tablet 26ad523451b64159877795981558e17c (DEFAULT_TABLE table=rebalance_test_table_2 [id=be7f1e6a510b4bce8e966c8515685cff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.606525  8894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ad523451b64159877795981558e17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.608980  9590 tablet_bootstrap.cc:492] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:00.610888  9155 tablet_service.cc:1467] Processing CreateTablet for tablet 26ad523451b64159877795981558e17c (DEFAULT_TABLE table=rebalance_test_table_2 [id=be7f1e6a510b4bce8e966c8515685cff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.611230  9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ad523451b64159877795981558e17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.612709  9590 tablet_bootstrap.cc:654] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.612905  9591 tablet_bootstrap.cc:492] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:00.614604  9591 tablet_bootstrap.cc:654] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.621235  9590 tablet_bootstrap.cc:492] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:00.621376  9590 ts_tablet_manager.cc:1367] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.013s	user 0.002s	sys 0.003s
I20240501 20:09:00.622136  9590 raft_consensus.cc:348] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.622381  9590 raft_consensus.cc:374] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.622459  9590 raft_consensus.cc:729] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.624480  9590 consensus_queue.cc:260] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.625456  9590 ts_tablet_manager.cc:1398] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:09:00.632208  9590 tablet_bootstrap.cc:492] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:00.633913  9590 tablet_bootstrap.cc:654] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.635169  9597 tablet_bootstrap.cc:492] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:00.636135  9590 tablet_bootstrap.cc:492] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:00.636369  9590 ts_tablet_manager.cc:1367] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240501 20:09:00.636662  9597 tablet_bootstrap.cc:654] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.637153  9590 raft_consensus.cc:348] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:00.637439  9590 raft_consensus.cc:374] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.637562  9590 raft_consensus.cc:729] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.637708  9595 tablet_bootstrap.cc:492] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:00.637881  9590 consensus_queue.cc:260] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:00.638656  9590 ts_tablet_manager.cc:1398] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:09:00.639202  9595 tablet_bootstrap.cc:654] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.641664  9595 tablet_bootstrap.cc:492] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:00.641800  9595 ts_tablet_manager.cc:1367] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 20:09:00.642508  9595 raft_consensus.cc:348] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:00.642704  9595 raft_consensus.cc:374] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.642871  9595 raft_consensus.cc:729] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.643129  9595 consensus_queue.cc:260] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:00.643889  9589 tablet_bootstrap.cc:492] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:00.644766  9594 tablet_bootstrap.cc:492] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:00.644966  9591 tablet_bootstrap.cc:492] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:00.645100  9591 ts_tablet_manager.cc:1367] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.032s	user 0.011s	sys 0.015s
I20240501 20:09:00.645757  9591 raft_consensus.cc:348] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:00.645946  9591 raft_consensus.cc:374] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.646032  9591 raft_consensus.cc:729] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.646147  9589 tablet_bootstrap.cc:654] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.646265  9591 consensus_queue.cc:260] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:00.647722  9594 tablet_bootstrap.cc:654] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.648764  9591 ts_tablet_manager.cc:1398] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240501 20:09:00.649730  9595 ts_tablet_manager.cc:1398] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I20240501 20:09:00.652468  9594 tablet_bootstrap.cc:492] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:00.652590  9594 ts_tablet_manager.cc:1367] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240501 20:09:00.653195  9594 raft_consensus.cc:348] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:00.653350  9594 raft_consensus.cc:374] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.653438  9594 raft_consensus.cc:729] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.653620  9594 consensus_queue.cc:260] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:00.654029  9594 ts_tablet_manager.cc:1398] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240501 20:09:00.654312  9594 tablet_bootstrap.cc:492] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:00.655867  9594 tablet_bootstrap.cc:654] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.657263  9589 tablet_bootstrap.cc:492] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:00.657420  9589 ts_tablet_manager.cc:1367] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.000s
I20240501 20:09:00.657829  9597 tablet_bootstrap.cc:492] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:00.657969  9597 ts_tablet_manager.cc:1367] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.000s
I20240501 20:09:00.658113  9589 raft_consensus.cc:348] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.658262  9589 raft_consensus.cc:374] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.658370  9589 raft_consensus.cc:729] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.658550  9589 consensus_queue.cc:260] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.658625  9597 raft_consensus.cc:348] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:00.658843  9597 raft_consensus.cc:374] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.658931  9597 raft_consensus.cc:729] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.659168  9597 consensus_queue.cc:260] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:00.659243  9589 ts_tablet_manager.cc:1398] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:00.659480  9589 tablet_bootstrap.cc:492] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:00.660144  9597 ts_tablet_manager.cc:1398] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:00.660990  9589 tablet_bootstrap.cc:654] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.662775  9594 tablet_bootstrap.cc:492] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:00.662925  9594 ts_tablet_manager.cc:1367] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.003s
I20240501 20:09:00.663120  9589 tablet_bootstrap.cc:492] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:00.663273  9589 ts_tablet_manager.cc:1367] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:09:00.663587  9594 raft_consensus.cc:348] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.663784  9594 raft_consensus.cc:374] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.663861  9594 raft_consensus.cc:729] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.664052  9589 raft_consensus.cc:348] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:00.664302  9589 raft_consensus.cc:374] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.664067  9594 consensus_queue.cc:260] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.664404  9589 raft_consensus.cc:729] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.664631  9589 consensus_queue.cc:260] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:00.664729  9594 ts_tablet_manager.cc:1398] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:09:00.665056  9589 ts_tablet_manager.cc:1398] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:00.713783  9579 raft_consensus.cc:482] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:00.713923  9579 raft_consensus.cc:504] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:00.714514  9579 leader_election.cc:288] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:00.714944  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc72717208547689b45b9c789c8f3f9" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:00.715040  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc72717208547689b45b9c789c8f3f9" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:00.715137  9044 raft_consensus.cc:2454] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:00.715231  9306 raft_consensus.cc:2454] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:00.715565  8717 leader_election.cc:302] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:00.715802  9579 raft_consensus.cc:2781] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:00.715905  9579 raft_consensus.cc:482] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:00.715986  9579 raft_consensus.cc:3037] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.716854  9604 raft_consensus.cc:482] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:00.717000  9604 raft_consensus.cc:504] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.718189  9579 raft_consensus.cc:504] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:00.718309  9522 raft_consensus.cc:482] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:00.718493  9522 raft_consensus.cc:504] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.718796  9579 leader_election.cc:288] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:00.719143  9522 leader_election.cc:288] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:00.719471  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc72717208547689b45b9c789c8f3f9" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:00.719640  9306 raft_consensus.cc:3037] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.720024  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4e4fa3e651453ba0344d5dc2607700" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:00.720207  9176 raft_consensus.cc:2454] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:00.720590  8849 leader_election.cc:302] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:00.721225  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4e4fa3e651453ba0344d5dc2607700" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:00.721426  8652 raft_consensus.cc:2454] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:00.721999  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc72717208547689b45b9c789c8f3f9" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:00.722146  9306 raft_consensus.cc:2454] T 3cc72717208547689b45b9c789c8f3f9 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:00.722169  9044 raft_consensus.cc:3037] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.722335  9522 raft_consensus.cc:2781] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:00.722429  9522 raft_consensus.cc:482] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:00.722417  9604 leader_election.cc:288] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:00.722534  9522 raft_consensus.cc:3037] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.724725  9044 raft_consensus.cc:2454] T 3cc72717208547689b45b9c789c8f3f9 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:00.725103  8717 leader_election.cc:302] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:00.725229  9522 raft_consensus.cc:504] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.725390  9579 raft_consensus.cc:2781] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:00.726243  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4e4fa3e651453ba0344d5dc2607700" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:00.726457  9176 raft_consensus.cc:3037] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.726711  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4e4fa3e651453ba0344d5dc2607700" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:00.726918  8652 raft_consensus.cc:3037] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.729326  9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4e4fa3e651453ba0344d5dc2607700" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:00.729951  9176 raft_consensus.cc:2454] T 6b4e4fa3e651453ba0344d5dc2607700 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:00.730405  8849 leader_election.cc:302] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:00.730880  8652 raft_consensus.cc:2454] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:00.731379  9522 leader_election.cc:288] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Requested vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:00.731365  9579 raft_consensus.cc:686] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:09:00.731536  9522 raft_consensus.cc:2781] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:00.731707  9522 raft_consensus.cc:686] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Running, Role: LEADER
I20240501 20:09:00.731688  9579 consensus_queue.cc:237] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:00.731968  9522 consensus_queue.cc:237] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:00.733180  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4e4fa3e651453ba0344d5dc2607700" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:09:00.733768  8585 leader_election.cc:302] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225
I20240501 20:09:00.734470  9604 raft_consensus.cc:2726] T 6b4e4fa3e651453ba0344d5dc2607700 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:00.735730  8508 catalog_manager.cc:5484] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fb13c94e664ec280fad2968e515d19 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "02fb13c94e664ec280fad2968e515d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:00.735730  8506 catalog_manager.cc:5484] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:00.949631  9522 raft_consensus.cc:482] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:00.949800  9522 raft_consensus.cc:504] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:00.950444  9522 leader_election.cc:288] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:00.950829  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ad523451b64159877795981558e17c" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:00.950893  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ad523451b64159877795981558e17c" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:00.951022  8652 raft_consensus.cc:2454] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:00.951108  9176 raft_consensus.cc:2454] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:00.951380  8848 leader_election.cc:302] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:00.951670  9522 raft_consensus.cc:2781] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:00.951764  9522 raft_consensus.cc:482] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:00.951854  9522 raft_consensus.cc:3037] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.953589  9522 raft_consensus.cc:504] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:00.954051  9522 leader_election.cc:288] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Requested vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:00.954439  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ad523451b64159877795981558e17c" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:00.954461  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ad523451b64159877795981558e17c" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:00.954597  8652 raft_consensus.cc:3037] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.954640  9176 raft_consensus.cc:3037] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.956310  8652 raft_consensus.cc:2454] T 26ad523451b64159877795981558e17c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:00.956391  9176 raft_consensus.cc:2454] T 26ad523451b64159877795981558e17c P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:00.956656  8848 leader_election.cc:302] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:00.956905  9522 raft_consensus.cc:2781] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:00.957026  9522 raft_consensus.cc:686] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Running, Role: LEADER
I20240501 20:09:00.957285  9522 consensus_queue.cc:237] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:00.959640  8506 catalog_manager.cc:5484] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fb13c94e664ec280fad2968e515d19 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "02fb13c94e664ec280fad2968e515d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:01.115500  9579 consensus_queue.cc:1035] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:01.122555  9610 consensus_queue.cc:1035] T 3cc72717208547689b45b9c789c8f3f9 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:01.290239  9615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:01.290630  9615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:09:01.298413  9522 consensus_queue.cc:1035] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:01.299639  9615 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:01.309216  9611 consensus_queue.cc:1035] T 6b4e4fa3e651453ba0344d5dc2607700 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 20:09:01.358165  9628 consensus_queue.cc:1035] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:01.384303  9628 consensus_queue.cc:1035] T 26ad523451b64159877795981558e17c P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 223.245 ms
  time per row: 0.111622 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:09:02.533145  8508 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:02.541539  9025 tablet_service.cc:1467] Processing CreateTablet for tablet 863b82c7886746d2a3932c9cc6ce9019 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b027f19756aa4c5d82a18d0bb1f55710]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.541664  9287 tablet_service.cc:1467] Processing CreateTablet for tablet 863b82c7886746d2a3932c9cc6ce9019 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b027f19756aa4c5d82a18d0bb1f55710]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.542012  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863b82c7886746d2a3932c9cc6ce9019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.542089  9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863b82c7886746d2a3932c9cc6ce9019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.542838  8763 tablet_service.cc:1467] Processing CreateTablet for tablet 42a721314bd4471eb2ffe27dc8acdf5c (DEFAULT_TABLE table=rebalance_test_table_3 [id=b027f19756aa4c5d82a18d0bb1f55710]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.543210  8763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a721314bd4471eb2ffe27dc8acdf5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.546643  9687 tablet_bootstrap.cc:492] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:02.546576  8632 tablet_service.cc:1467] Processing CreateTablet for tablet dd0b0c1f3bf0404e99a6ea9c73bef90e (DEFAULT_TABLE table=rebalance_test_table_3 [id=b027f19756aa4c5d82a18d0bb1f55710]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.546636  8762 tablet_service.cc:1467] Processing CreateTablet for tablet 863b82c7886746d2a3932c9cc6ce9019 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b027f19756aa4c5d82a18d0bb1f55710]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.546955  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0b0c1f3bf0404e99a6ea9c73bef90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.546993  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863b82c7886746d2a3932c9cc6ce9019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.548244  9687 tablet_bootstrap.cc:654] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.550657  8894 tablet_service.cc:1467] Processing CreateTablet for tablet dd0b0c1f3bf0404e99a6ea9c73bef90e (DEFAULT_TABLE table=rebalance_test_table_3 [id=b027f19756aa4c5d82a18d0bb1f55710]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.551095  8894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0b0c1f3bf0404e99a6ea9c73bef90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.554839  9155 tablet_service.cc:1467] Processing CreateTablet for tablet dd0b0c1f3bf0404e99a6ea9c73bef90e (DEFAULT_TABLE table=rebalance_test_table_3 [id=b027f19756aa4c5d82a18d0bb1f55710]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.554857  9286 tablet_service.cc:1467] Processing CreateTablet for tablet 42a721314bd4471eb2ffe27dc8acdf5c (DEFAULT_TABLE table=rebalance_test_table_3 [id=b027f19756aa4c5d82a18d0bb1f55710]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.555193  9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a721314bd4471eb2ffe27dc8acdf5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.555212  9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0b0c1f3bf0404e99a6ea9c73bef90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.558876  9024 tablet_service.cc:1467] Processing CreateTablet for tablet 42a721314bd4471eb2ffe27dc8acdf5c (DEFAULT_TABLE table=rebalance_test_table_3 [id=b027f19756aa4c5d82a18d0bb1f55710]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.559231  9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a721314bd4471eb2ffe27dc8acdf5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.560200  9690 tablet_bootstrap.cc:492] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:02.561774  9690 tablet_bootstrap.cc:654] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.564096  9687 tablet_bootstrap.cc:492] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:02.564234  9687 ts_tablet_manager.cc:1367] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.018s	user 0.000s	sys 0.017s
I20240501 20:09:02.564924  9687 raft_consensus.cc:348] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:02.565121  9687 raft_consensus.cc:374] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.565205  9687 raft_consensus.cc:729] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.565399  9687 consensus_queue.cc:260] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:02.566008  9687 ts_tablet_manager.cc:1398] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:09:02.566252  9687 tablet_bootstrap.cc:492] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:02.567603  9687 tablet_bootstrap.cc:654] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.570860  9694 tablet_bootstrap.cc:492] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:02.571858  9693 tablet_bootstrap.cc:492] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:02.572300  9694 tablet_bootstrap.cc:654] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.573164  9692 tablet_bootstrap.cc:492] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:02.573323  9693 tablet_bootstrap.cc:654] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.574337  9687 tablet_bootstrap.cc:492] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:02.574486  9687 ts_tablet_manager.cc:1367] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240501 20:09:02.574880  9692 tablet_bootstrap.cc:654] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.575119  9687 raft_consensus.cc:348] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:02.575335  9687 raft_consensus.cc:374] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.575405  9687 raft_consensus.cc:729] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.575485  9688 tablet_bootstrap.cc:492] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:02.575605  9687 consensus_queue.cc:260] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:02.576745  9687 ts_tablet_manager.cc:1398] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:02.576930  9688 tablet_bootstrap.cc:654] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.577885  9690 tablet_bootstrap.cc:492] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:02.578023  9690 ts_tablet_manager.cc:1367] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.000s
I20240501 20:09:02.579376  9690 raft_consensus.cc:348] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:02.579586  9690 raft_consensus.cc:374] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.579660  9690 raft_consensus.cc:729] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.579886  9690 consensus_queue.cc:260] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:02.581768  9694 tablet_bootstrap.cc:492] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:02.581936  9694 ts_tablet_manager.cc:1367] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.004s
I20240501 20:09:02.582983  9694 raft_consensus.cc:348] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:02.583246  9694 raft_consensus.cc:374] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.583384  9694 raft_consensus.cc:729] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.583473  9693 tablet_bootstrap.cc:492] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:02.583624  9693 ts_tablet_manager.cc:1367] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.012s	user 0.000s	sys 0.006s
I20240501 20:09:02.583745  9694 consensus_queue.cc:260] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:02.584332  9693 raft_consensus.cc:348] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:02.584547  9693 raft_consensus.cc:374] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.584636  9693 raft_consensus.cc:729] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.585327  9692 tablet_bootstrap.cc:492] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:02.585533  9692 ts_tablet_manager.cc:1367] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.003s
I20240501 20:09:02.586333  9692 raft_consensus.cc:348] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:02.586623  9692 raft_consensus.cc:374] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.586750  9692 raft_consensus.cc:729] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.587030  9692 consensus_queue.cc:260] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:02.588232  9692 ts_tablet_manager.cc:1398] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240501 20:09:02.589552  9690 ts_tablet_manager.cc:1398] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.006s
I20240501 20:09:02.590051  9693 consensus_queue.cc:260] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:02.590976  9693 ts_tablet_manager.cc:1398] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240501 20:09:02.592448  9688 tablet_bootstrap.cc:492] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:02.592635  9688 ts_tablet_manager.cc:1367] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.008s
I20240501 20:09:02.593403  9688 raft_consensus.cc:348] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:02.593629  9688 raft_consensus.cc:374] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.593763  9688 raft_consensus.cc:729] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.593950  9694 ts_tablet_manager.cc:1398] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.004s
I20240501 20:09:02.594077  9688 consensus_queue.cc:260] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:02.594228  9694 tablet_bootstrap.cc:492] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:02.594684  9688 ts_tablet_manager.cc:1398] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:02.594928  9688 tablet_bootstrap.cc:492] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:02.595757  9694 tablet_bootstrap.cc:654] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.596122  9688 tablet_bootstrap.cc:654] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.597893  9694 tablet_bootstrap.cc:492] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:02.598042  9694 ts_tablet_manager.cc:1367] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240501 20:09:02.598822  9694 raft_consensus.cc:348] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:02.599042  9694 raft_consensus.cc:374] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.599113  9694 raft_consensus.cc:729] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.599356  9694 consensus_queue.cc:260] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:02.599934  9694 ts_tablet_manager.cc:1398] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:02.600570  9688 tablet_bootstrap.cc:492] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:02.600700  9688 ts_tablet_manager.cc:1367] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240501 20:09:02.601248  9688 raft_consensus.cc:348] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:02.601426  9688 raft_consensus.cc:374] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.601498  9688 raft_consensus.cc:729] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.601706  9688 consensus_queue.cc:260] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:02.602121  9688 ts_tablet_manager.cc:1398] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:09:02.679930  9522 raft_consensus.cc:482] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:02.680069  9522 raft_consensus.cc:504] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:02.680567  9522 leader_election.cc:288] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:02.680994  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0b0c1f3bf0404e99a6ea9c73bef90e" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:02.681036  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0b0c1f3bf0404e99a6ea9c73bef90e" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:02.681226  8652 raft_consensus.cc:2454] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:02.681273  9176 raft_consensus.cc:2454] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:02.681650  8848 leader_election.cc:302] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:02.681946  9522 raft_consensus.cc:2781] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:02.682040  9522 raft_consensus.cc:482] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:02.682123  9522 raft_consensus.cc:3037] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:02.683936  9522 raft_consensus.cc:504] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:02.684433  9522 leader_election.cc:288] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Requested vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:02.684727  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0b0c1f3bf0404e99a6ea9c73bef90e" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:02.684850  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0b0c1f3bf0404e99a6ea9c73bef90e" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:02.684922  8652 raft_consensus.cc:3037] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:02.685021  9176 raft_consensus.cc:3037] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:02.685051  9580 raft_consensus.cc:482] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:02.685178  9580 raft_consensus.cc:504] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:02.685825  9580 leader_election.cc:288] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:02.686223  8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a721314bd4471eb2ffe27dc8acdf5c" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" is_pre_election: true
I20240501 20:09:02.686412  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a721314bd4471eb2ffe27dc8acdf5c" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:02.686486  8782 raft_consensus.cc:2454] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:09:02.686615  9044 raft_consensus.cc:2454] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:09:02.686901  9240 leader_election.cc:302] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:02.687168  8652 raft_consensus.cc:2454] T dd0b0c1f3bf0404e99a6ea9c73bef90e P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:02.687203  9580 raft_consensus.cc:2781] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:02.687309  9580 raft_consensus.cc:482] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:02.687386  9580 raft_consensus.cc:3037] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:02.687547  8848 leader_election.cc:302] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:02.687590  9176 raft_consensus.cc:2454] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:02.687773  9522 raft_consensus.cc:2781] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:02.688158  9522 raft_consensus.cc:686] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Running, Role: LEADER
I20240501 20:09:02.688496  9522 consensus_queue.cc:237] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:02.689330  9580 raft_consensus.cc:504] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:02.690318  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a721314bd4471eb2ffe27dc8acdf5c" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:02.690496  9044 raft_consensus.cc:3037] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:02.690467  8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a721314bd4471eb2ffe27dc8acdf5c" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
I20240501 20:09:02.690620  8782 raft_consensus.cc:3037] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:02.691265  8505 catalog_manager.cc:5484] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fb13c94e664ec280fad2968e515d19 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "02fb13c94e664ec280fad2968e515d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:02.692441  8782 raft_consensus.cc:2454] T 42a721314bd4471eb2ffe27dc8acdf5c P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:09:02.692816  9044 raft_consensus.cc:2454] T 42a721314bd4471eb2ffe27dc8acdf5c P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:09:02.692812  9240 leader_election.cc:302] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:02.693323  9708 raft_consensus.cc:2781] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:02.693513  9580 leader_election.cc:288] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:02.693955  9708 raft_consensus.cc:686] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [term 1 LEADER]: Becoming Leader. State: Replica: 23543295bda14131a70bd31c573454f6, State: Running, Role: LEADER
I20240501 20:09:02.694209  9708 consensus_queue.cc:237] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:02.696321  8505 catalog_manager.cc:5484] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23543295bda14131a70bd31c573454f6 (127.8.68.198). New cstate: current_term: 1 leader_uuid: "23543295bda14131a70bd31c573454f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:02.791930  9708 raft_consensus.cc:482] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:02.792122  9708 raft_consensus.cc:504] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:02.792722  9708 leader_election.cc:288] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923)
I20240501 20:09:02.793021  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863b82c7886746d2a3932c9cc6ce9019" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:02.793229  9044 raft_consensus.cc:2454] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:09:02.793658  9241 leader_election.cc:302] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3b6c3b236a5d40bf99869152603b5524; no voters: 
I20240501 20:09:02.793753  8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863b82c7886746d2a3932c9cc6ce9019" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" is_pre_election: true
I20240501 20:09:02.793977  8782 raft_consensus.cc:2454] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:09:02.794047  9708 raft_consensus.cc:2781] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:02.794124  9708 raft_consensus.cc:482] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:02.794224  9708 raft_consensus.cc:3037] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:02.796016  9708 raft_consensus.cc:504] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:02.796454  9708 leader_election.cc:288] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923)
I20240501 20:09:02.796809  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863b82c7886746d2a3932c9cc6ce9019" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:02.796875  8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863b82c7886746d2a3932c9cc6ce9019" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
I20240501 20:09:02.796970  9044 raft_consensus.cc:3037] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:02.797075  8782 raft_consensus.cc:3037] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:02.798882  9044 raft_consensus.cc:2454] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:09:02.799315  8782 raft_consensus.cc:2454] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:09:02.799316  9241 leader_election.cc:302] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3b6c3b236a5d40bf99869152603b5524; no voters: 
I20240501 20:09:02.799549  9708 raft_consensus.cc:2781] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:02.799723  9708 raft_consensus.cc:686] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [term 1 LEADER]: Becoming Leader. State: Replica: 23543295bda14131a70bd31c573454f6, State: Running, Role: LEADER
I20240501 20:09:02.800036  9708 consensus_queue.cc:237] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:02.802579  8505 catalog_manager.cc:5484] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23543295bda14131a70bd31c573454f6 (127.8.68.198). New cstate: current_term: 1 leader_uuid: "23543295bda14131a70bd31c573454f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:03.156456  9712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:03.156769  9712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:09:03.163659  9522 consensus_queue.cc:1035] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.165195  9706 consensus_queue.cc:1035] T dd0b0c1f3bf0404e99a6ea9c73bef90e P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:03.168917  9712 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:03.207659  9708 consensus_queue.cc:1035] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.214198  9709 consensus_queue.cc:1035] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 20:09:03.344946  8783 raft_consensus.cc:1261] T 863b82c7886746d2a3932c9cc6ce9019 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Refusing update from remote peer 23543295bda14131a70bd31c573454f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:03.346360  9709 consensus_queue.cc:1035] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.413580  9045 raft_consensus.cc:1261] T 863b82c7886746d2a3932c9cc6ce9019 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Refusing update from remote peer 23543295bda14131a70bd31c573454f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:03.414588  9709 consensus_queue.cc:1035] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.445837  9734 mvcc.cc:205] Tried to move back new op lower bound from 7022977611097018368 to 7022977608910970880. Current Snapshot: MvccSnapshot[applied={T|T < 7022977611097018368}]

INSERT report
    rows total: 2000
    time total: 254.247 ms
  time per row: 0.127124 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:09:04.440399  8504 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:04.449010  8894 tablet_service.cc:1467] Processing CreateTablet for tablet da583015bf3e4d3b99c32d653dc8131b (DEFAULT_TABLE table=rebalance_test_table_4 [id=e80581b7baf24e63b7443b07276af0de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.449249  8632 tablet_service.cc:1467] Processing CreateTablet for tablet da583015bf3e4d3b99c32d653dc8131b (DEFAULT_TABLE table=rebalance_test_table_4 [id=e80581b7baf24e63b7443b07276af0de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.449445  8894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da583015bf3e4d3b99c32d653dc8131b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.449602  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da583015bf3e4d3b99c32d653dc8131b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.450533  9024 tablet_service.cc:1467] Processing CreateTablet for tablet 9e288e6d1c404303a044cbae80e13491 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e80581b7baf24e63b7443b07276af0de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.450703  8763 tablet_service.cc:1467] Processing CreateTablet for tablet 9e288e6d1c404303a044cbae80e13491 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e80581b7baf24e63b7443b07276af0de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.450944  9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e288e6d1c404303a044cbae80e13491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.451081  8763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e288e6d1c404303a044cbae80e13491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.451253  8893 tablet_service.cc:1467] Processing CreateTablet for tablet f706a5f9c68a4a9e8cf7988e2ee71c63 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e80581b7baf24e63b7443b07276af0de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.451596  8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f706a5f9c68a4a9e8cf7988e2ee71c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.455086  9286 tablet_service.cc:1467] Processing CreateTablet for tablet 9e288e6d1c404303a044cbae80e13491 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e80581b7baf24e63b7443b07276af0de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.455473  9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e288e6d1c404303a044cbae80e13491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.450559  8631 tablet_service.cc:1467] Processing CreateTablet for tablet f706a5f9c68a4a9e8cf7988e2ee71c63 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e80581b7baf24e63b7443b07276af0de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.457065  8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f706a5f9c68a4a9e8cf7988e2ee71c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.459897  9155 tablet_service.cc:1467] Processing CreateTablet for tablet da583015bf3e4d3b99c32d653dc8131b (DEFAULT_TABLE table=rebalance_test_table_4 [id=e80581b7baf24e63b7443b07276af0de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.459972  9156 tablet_service.cc:1467] Processing CreateTablet for tablet f706a5f9c68a4a9e8cf7988e2ee71c63 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e80581b7baf24e63b7443b07276af0de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.460284  9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da583015bf3e4d3b99c32d653dc8131b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.460630  9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f706a5f9c68a4a9e8cf7988e2ee71c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.466179  9780 tablet_bootstrap.cc:492] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:04.467878  9780 tablet_bootstrap.cc:654] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.470889  9785 tablet_bootstrap.cc:492] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:04.473587  9785 tablet_bootstrap.cc:654] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.474423  9783 tablet_bootstrap.cc:492] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:04.477823  9781 tablet_bootstrap.cc:492] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:04.478829  9782 tablet_bootstrap.cc:492] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:04.480275  9782 tablet_bootstrap.cc:654] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.480465  9783 tablet_bootstrap.cc:654] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.481479  9779 tablet_bootstrap.cc:492] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:04.482230  9781 tablet_bootstrap.cc:654] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.484279  9779 tablet_bootstrap.cc:654] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.501899  9783 tablet_bootstrap.cc:492] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:04.502038  9783 ts_tablet_manager.cc:1367] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.028s	user 0.003s	sys 0.003s
I20240501 20:09:04.504217  9782 tablet_bootstrap.cc:492] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:04.504423  9782 ts_tablet_manager.cc:1367] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.026s	user 0.002s	sys 0.010s
I20240501 20:09:04.503567  9780 tablet_bootstrap.cc:492] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:04.504853  9780 ts_tablet_manager.cc:1367] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.039s	user 0.014s	sys 0.006s
I20240501 20:09:04.505162  9782 raft_consensus.cc:348] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:04.505447  9782 raft_consensus.cc:374] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.505601  9782 raft_consensus.cc:729] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.505697  9780 raft_consensus.cc:348] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:04.505858  9782 consensus_queue.cc:260] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:04.506085  9780 raft_consensus.cc:374] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.506227  9780 raft_consensus.cc:729] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.506472  9782 ts_tablet_manager.cc:1398] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:04.506518  9780 consensus_queue.cc:260] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:04.506811  9782 tablet_bootstrap.cc:492] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:04.508219  9782 tablet_bootstrap.cc:654] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.508893  9785 tablet_bootstrap.cc:492] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:04.509105  9785 ts_tablet_manager.cc:1367] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.039s	user 0.011s	sys 0.006s
I20240501 20:09:04.509923  9785 raft_consensus.cc:348] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:04.510244  9785 raft_consensus.cc:374] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.510412  9785 raft_consensus.cc:729] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.504704  9783 raft_consensus.cc:348] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:04.510675  9783 raft_consensus.cc:374] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.510661  9785 consensus_queue.cc:260] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:04.510846  9783 raft_consensus.cc:729] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.511250  9785 ts_tablet_manager.cc:1398] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:04.511281  9783 consensus_queue.cc:260] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:04.513648  9783 ts_tablet_manager.cc:1398] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.001s
I20240501 20:09:04.516590  9783 tablet_bootstrap.cc:492] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:04.521323  9779 tablet_bootstrap.cc:492] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:04.521457  9779 ts_tablet_manager.cc:1367] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.040s	user 0.002s	sys 0.010s
I20240501 20:09:04.521553  9783 tablet_bootstrap.cc:654] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.522068  9779 raft_consensus.cc:348] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:04.522243  9779 raft_consensus.cc:374] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.522397  9779 raft_consensus.cc:729] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.522715  9779 consensus_queue.cc:260] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:04.523128  9781 tablet_bootstrap.cc:492] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:04.523257  9781 ts_tablet_manager.cc:1367] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.046s	user 0.003s	sys 0.004s
I20240501 20:09:04.523262  9792 raft_consensus.cc:482] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:04.523376  9780 ts_tablet_manager.cc:1398] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.009s
I20240501 20:09:04.523414  9792 raft_consensus.cc:504] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:04.524317  9781 raft_consensus.cc:348] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:04.524525  9781 raft_consensus.cc:374] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.524602  9781 raft_consensus.cc:729] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.524844  9781 consensus_queue.cc:260] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:04.525218  9792 leader_election.cc:288] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:04.527683  9783 tablet_bootstrap.cc:492] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:04.527894  9783 ts_tablet_manager.cc:1367] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240501 20:09:04.528586  9783 raft_consensus.cc:348] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:04.528843  9783 raft_consensus.cc:374] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.528949  9783 raft_consensus.cc:729] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.529206  9783 consensus_queue.cc:260] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:04.529681  9783 ts_tablet_manager.cc:1398] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:04.530997  9782 tablet_bootstrap.cc:492] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:04.531347  9782 ts_tablet_manager.cc:1367] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.025s	user 0.005s	sys 0.000s
I20240501 20:09:04.532061  9782 raft_consensus.cc:348] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:04.532321  9782 raft_consensus.cc:374] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.532436  9782 raft_consensus.cc:729] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.532706  9782 consensus_queue.cc:260] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:04.534024  9782 ts_tablet_manager.cc:1398] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:04.537665  9781 ts_tablet_manager.cc:1398] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.014s	user 0.001s	sys 0.004s
I20240501 20:09:04.540438  9779 ts_tablet_manager.cc:1398] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.019s	user 0.003s	sys 0.005s
I20240501 20:09:04.541237  9779 tablet_bootstrap.cc:492] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:04.542024  8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e288e6d1c404303a044cbae80e13491" candidate_uuid: "3b6c3b236a5d40bf99869152603b5524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" is_pre_election: true
I20240501 20:09:04.542217  8782 raft_consensus.cc:2454] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6c3b236a5d40bf99869152603b5524 in term 0.
I20240501 20:09:04.542707  9779 tablet_bootstrap.cc:654] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.542755  8978 leader_election.cc:302] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:04.543184  9794 raft_consensus.cc:2781] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:04.543277  9794 raft_consensus.cc:482] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:04.543394  9794 raft_consensus.cc:3037] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.546545  9794 raft_consensus.cc:504] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:04.547304  9794 leader_election.cc:288] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:04.547745  8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e288e6d1c404303a044cbae80e13491" candidate_uuid: "3b6c3b236a5d40bf99869152603b5524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
I20240501 20:09:04.547938  8782 raft_consensus.cc:3037] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.547945  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e288e6d1c404303a044cbae80e13491" candidate_uuid: "3b6c3b236a5d40bf99869152603b5524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:04.548126  9307 raft_consensus.cc:3037] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.549609  9779 tablet_bootstrap.cc:492] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:04.549750  9779 ts_tablet_manager.cc:1367] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.001s
I20240501 20:09:04.550108  8782 raft_consensus.cc:2454] T 9e288e6d1c404303a044cbae80e13491 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6c3b236a5d40bf99869152603b5524 in term 1.
I20240501 20:09:04.550500  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e288e6d1c404303a044cbae80e13491" candidate_uuid: "3b6c3b236a5d40bf99869152603b5524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:04.550547  8978 leader_election.cc:302] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:04.550500  9779 raft_consensus.cc:348] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:04.550709  9779 raft_consensus.cc:374] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.550777  9779 raft_consensus.cc:729] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.550792  9307 raft_consensus.cc:2454] T 9e288e6d1c404303a044cbae80e13491 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6c3b236a5d40bf99869152603b5524 in term 1.
I20240501 20:09:04.551056  9779 consensus_queue.cc:260] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:04.551524  9779 ts_tablet_manager.cc:1398] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:04.551824  9794 raft_consensus.cc:2781] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:04.552007  9794 raft_consensus.cc:686] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [term 1 LEADER]: Becoming Leader. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Running, Role: LEADER
I20240501 20:09:04.552577  9794 consensus_queue.cc:237] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:04.555517  8508 catalog_manager.cc:5484] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196). New cstate: current_term: 1 leader_uuid: "3b6c3b236a5d40bf99869152603b5524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:04.573937  9417 raft_consensus.cc:482] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:04.574062  9417 raft_consensus.cc:504] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:04.574654  9417 leader_election.cc:288] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647)
I20240501 20:09:04.575125  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f706a5f9c68a4a9e8cf7988e2ee71c63" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:09:04.575114  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f706a5f9c68a4a9e8cf7988e2ee71c63" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:04.575356  8913 raft_consensus.cc:2454] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 0.
I20240501 20:09:04.575337  8652 raft_consensus.cc:2454] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 0.
I20240501 20:09:04.575752  9110 leader_election.cc:302] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea5761ef47546768883016105c52225, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:04.576053  9417 raft_consensus.cc:2781] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:04.576159  9417 raft_consensus.cc:482] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:04.576257  9417 raft_consensus.cc:3037] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.578001  9417 raft_consensus.cc:504] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:04.578548  9417 leader_election.cc:288] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 election: Requested vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647)
I20240501 20:09:04.578959  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f706a5f9c68a4a9e8cf7988e2ee71c63" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19"
I20240501 20:09:04.578945  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f706a5f9c68a4a9e8cf7988e2ee71c63" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:04.579131  8652 raft_consensus.cc:3037] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.579152  8913 raft_consensus.cc:3037] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.581821  8913 raft_consensus.cc:2454] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 1.
I20240501 20:09:04.581840  8652 raft_consensus.cc:2454] T f706a5f9c68a4a9e8cf7988e2ee71c63 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 1.
I20240501 20:09:04.582221  9109 leader_election.cc:302] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:04.582526  9417 raft_consensus.cc:2781] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:04.582870  9417 raft_consensus.cc:686] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea5761ef47546768883016105c52225, State: Running, Role: LEADER
I20240501 20:09:04.583141  9417 consensus_queue.cc:237] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:04.585486  8508 catalog_manager.cc:5484] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ea5761ef47546768883016105c52225 (127.8.68.197). New cstate: current_term: 1 leader_uuid: "4ea5761ef47546768883016105c52225" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:04.676885  9417 raft_consensus.cc:482] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:04.677079  9417 raft_consensus.cc:504] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:04.677744  9417 leader_election.cc:288] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:04.677999  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da583015bf3e4d3b99c32d653dc8131b" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:09:04.678164  8913 raft_consensus.cc:2454] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 0.
I20240501 20:09:04.678125  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da583015bf3e4d3b99c32d653dc8131b" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:04.678395  8652 raft_consensus.cc:2454] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 0.
I20240501 20:09:04.678531  9109 leader_election.cc:302] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:04.678798  9417 raft_consensus.cc:2781] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:04.678889  9417 raft_consensus.cc:482] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:04.678989  9417 raft_consensus.cc:3037] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.680684  9417 raft_consensus.cc:504] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:04.681182  9417 leader_election.cc:288] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 election: Requested vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:04.681522  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da583015bf3e4d3b99c32d653dc8131b" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:04.681566  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da583015bf3e4d3b99c32d653dc8131b" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19"
I20240501 20:09:04.681710  8652 raft_consensus.cc:3037] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.681733  8913 raft_consensus.cc:3037] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.683547  8913 raft_consensus.cc:2454] T da583015bf3e4d3b99c32d653dc8131b P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 1.
I20240501 20:09:04.683598  8652 raft_consensus.cc:2454] T da583015bf3e4d3b99c32d653dc8131b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 1.
I20240501 20:09:04.683893  9109 leader_election.cc:302] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:04.684165  9417 raft_consensus.cc:2781] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:04.684343  9417 raft_consensus.cc:686] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea5761ef47546768883016105c52225, State: Running, Role: LEADER
I20240501 20:09:04.684587  9417 consensus_queue.cc:237] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:04.686923  8508 catalog_manager.cc:5484] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ea5761ef47546768883016105c52225 (127.8.68.197). New cstate: current_term: 1 leader_uuid: "4ea5761ef47546768883016105c52225" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:04.988176  9417 consensus_queue.cc:1035] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:04.992506  9794 consensus_queue.cc:1035] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:04.997534  9804 consensus_queue.cc:1035] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.004766  9792 consensus_queue.cc:1035] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:05.018151  9807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:05.018487  9807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:05.027155  9807 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 20:09:05.169003  9810 consensus_queue.cc:1035] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.187855  9417 consensus_queue.cc:1035] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 339.288 ms
  time per row: 0.169644 ms
I20240501 20:09:05.530514  8508 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:05.538897  9024 tablet_service.cc:1467] Processing CreateTablet for tablet 43bf7725b0b94c2eac42fca017b10a2f (DEFAULT_TABLE table=rebalance_test_table_5 [id=dde26843bd154b80b3d148d7e5accfd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.539387  9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bf7725b0b94c2eac42fca017b10a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.539512  9025 tablet_service.cc:1467] Processing CreateTablet for tablet d205586c09e8445f91ea63579e8b8ded (DEFAULT_TABLE table=rebalance_test_table_5 [id=dde26843bd154b80b3d148d7e5accfd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.539845  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d205586c09e8445f91ea63579e8b8ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.540261  9156 tablet_service.cc:1467] Processing CreateTablet for tablet dd5acbf391ca4f7281677b2f2584a7f9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dde26843bd154b80b3d148d7e5accfd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.540663  9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5acbf391ca4f7281677b2f2584a7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.541266  8763 tablet_service.cc:1467] Processing CreateTablet for tablet 43bf7725b0b94c2eac42fca017b10a2f (DEFAULT_TABLE table=rebalance_test_table_5 [id=dde26843bd154b80b3d148d7e5accfd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.541651  8763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bf7725b0b94c2eac42fca017b10a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.543138  8893 tablet_service.cc:1467] Processing CreateTablet for tablet dd5acbf391ca4f7281677b2f2584a7f9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dde26843bd154b80b3d148d7e5accfd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.543524  8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5acbf391ca4f7281677b2f2584a7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.546681  8762 tablet_service.cc:1467] Processing CreateTablet for tablet d205586c09e8445f91ea63579e8b8ded (DEFAULT_TABLE table=rebalance_test_table_5 [id=dde26843bd154b80b3d148d7e5accfd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.547004  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d205586c09e8445f91ea63579e8b8ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.550652  9286 tablet_service.cc:1467] Processing CreateTablet for tablet 43bf7725b0b94c2eac42fca017b10a2f (DEFAULT_TABLE table=rebalance_test_table_5 [id=dde26843bd154b80b3d148d7e5accfd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.551040  9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bf7725b0b94c2eac42fca017b10a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.551724  9874 tablet_bootstrap.cc:492] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:05.554606  9287 tablet_service.cc:1467] Processing CreateTablet for tablet d205586c09e8445f91ea63579e8b8ded (DEFAULT_TABLE table=rebalance_test_table_5 [id=dde26843bd154b80b3d148d7e5accfd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.554873  9872 tablet_bootstrap.cc:492] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:05.554911  9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d205586c09e8445f91ea63579e8b8ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.555492  9874 tablet_bootstrap.cc:654] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.556464  9872 tablet_bootstrap.cc:654] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.559217  9874 tablet_bootstrap.cc:492] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:05.559785  8631 tablet_service.cc:1467] Processing CreateTablet for tablet dd5acbf391ca4f7281677b2f2584a7f9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dde26843bd154b80b3d148d7e5accfd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:05.560173  8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5acbf391ca4f7281677b2f2584a7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:05.561725  9877 tablet_bootstrap.cc:492] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:05.563544  9877 tablet_bootstrap.cc:654] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.564424  9872 tablet_bootstrap.cc:492] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:05.564601  9872 ts_tablet_manager.cc:1367] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.006s
I20240501 20:09:05.565343  9872 raft_consensus.cc:348] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:05.565599  9872 raft_consensus.cc:374] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.565730  9872 raft_consensus.cc:729] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.566027  9872 consensus_queue.cc:260] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:05.566656  9872 ts_tablet_manager.cc:1398] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:05.567453  9872 tablet_bootstrap.cc:492] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:05.568121  9873 tablet_bootstrap.cc:492] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:05.569310  9872 tablet_bootstrap.cc:654] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.569784  9873 tablet_bootstrap.cc:654] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.570813  9880 tablet_bootstrap.cc:492] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:05.572928  9877 tablet_bootstrap.cc:492] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:05.573115  9877 ts_tablet_manager.cc:1367] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.003s
I20240501 20:09:05.573124  9874 ts_tablet_manager.cc:1367] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.000s
I20240501 20:09:05.573843  9877 raft_consensus.cc:348] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:05.573836  9874 raft_consensus.cc:348] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:05.574019  9880 tablet_bootstrap.cc:654] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.574029  9874 raft_consensus.cc:374] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.574112  9874 raft_consensus.cc:729] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.574182  9877 raft_consensus.cc:374] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.574347  9877 raft_consensus.cc:729] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.574364  9874 consensus_queue.cc:260] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:05.574538  9879 tablet_bootstrap.cc:492] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:05.574705  9877 consensus_queue.cc:260] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:05.574860  9874 ts_tablet_manager.cc:1398] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:05.575395  9877 ts_tablet_manager.cc:1398] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:05.575697  9877 tablet_bootstrap.cc:492] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:05.577188  9877 tablet_bootstrap.cc:654] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.578536  9879 tablet_bootstrap.cc:654] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.581849  9873 tablet_bootstrap.cc:492] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:05.581992  9873 ts_tablet_manager.cc:1367] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.000s
I20240501 20:09:05.583329  9873 raft_consensus.cc:348] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:05.583546  9873 raft_consensus.cc:374] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.583639  9873 raft_consensus.cc:729] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.583858  9873 consensus_queue.cc:260] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:05.584524  9873 ts_tablet_manager.cc:1398] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:05.585155  9880 tablet_bootstrap.cc:492] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:05.585289  9880 ts_tablet_manager.cc:1367] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.018s	user 0.003s	sys 0.000s
I20240501 20:09:05.586007  9880 raft_consensus.cc:348] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:05.586200  9880 raft_consensus.cc:374] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.586308  9880 raft_consensus.cc:729] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.586503  9880 consensus_queue.cc:260] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:05.587082  9872 tablet_bootstrap.cc:492] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:05.587219  9872 ts_tablet_manager.cc:1367] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.008s
I20240501 20:09:05.587797  9872 raft_consensus.cc:348] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:05.587966  9872 raft_consensus.cc:374] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.588039  9872 raft_consensus.cc:729] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.588259  9872 consensus_queue.cc:260] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:05.588620  9872 ts_tablet_manager.cc:1398] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:09:05.589509  9880 ts_tablet_manager.cc:1398] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240501 20:09:05.590785  9880 tablet_bootstrap.cc:492] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:05.593628  9879 tablet_bootstrap.cc:492] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:05.593735  9879 ts_tablet_manager.cc:1367] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.019s	user 0.000s	sys 0.003s
I20240501 20:09:05.594247  9879 raft_consensus.cc:348] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:05.594444  9879 raft_consensus.cc:374] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.594519  9880 tablet_bootstrap.cc:654] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:05.594534  9879 raft_consensus.cc:729] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.594781  9879 consensus_queue.cc:260] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:05.595291  9879 ts_tablet_manager.cc:1398] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.001s
I20240501 20:09:05.596560  9877 tablet_bootstrap.cc:492] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:05.597050  9877 ts_tablet_manager.cc:1367] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.013s
I20240501 20:09:05.597210  9880 tablet_bootstrap.cc:492] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:05.597407  9880 ts_tablet_manager.cc:1367] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.003s
I20240501 20:09:05.597766  9877 raft_consensus.cc:348] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:05.597977  9877 raft_consensus.cc:374] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.598058  9877 raft_consensus.cc:729] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.598167  9880 raft_consensus.cc:348] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:05.598425  9880 raft_consensus.cc:374] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:05.598335  9877 consensus_queue.cc:260] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:05.598510  9880 raft_consensus.cc:729] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:05.598852  9877 ts_tablet_manager.cc:1398] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:05.598836  9880 consensus_queue.cc:260] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:05.599361  9880 ts_tablet_manager.cc:1398] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:05.629634  9790 raft_consensus.cc:482] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:05.629757  9790 raft_consensus.cc:504] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:05.630484  9790 leader_election.cc:288] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:05.630918  9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5acbf391ca4f7281677b2f2584a7f9" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:05.630957  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5acbf391ca4f7281677b2f2584a7f9" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:05.631130  8652 raft_consensus.cc:2454] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:05.631143  9175 raft_consensus.cc:2454] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:05.631484  8848 leader_election.cc:302] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:05.631749  9790 raft_consensus.cc:2781] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:05.631840  9790 raft_consensus.cc:482] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:05.631920  9790 raft_consensus.cc:3037] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.633610  9790 raft_consensus.cc:504] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:05.634044  9790 leader_election.cc:288] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Requested vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:05.634358  9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5acbf391ca4f7281677b2f2584a7f9" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:05.634502  9175 raft_consensus.cc:3037] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.634470  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5acbf391ca4f7281677b2f2584a7f9" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:05.634632  8652 raft_consensus.cc:3037] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.636394  9175 raft_consensus.cc:2454] T dd5acbf391ca4f7281677b2f2584a7f9 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:05.636495  8652 raft_consensus.cc:2454] T dd5acbf391ca4f7281677b2f2584a7f9 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:05.636729  8849 leader_election.cc:302] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:05.637039  9790 raft_consensus.cc:2781] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:05.637387  9790 raft_consensus.cc:686] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Running, Role: LEADER
I20240501 20:09:05.637645  9790 consensus_queue.cc:237] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:05.639997  8504 catalog_manager.cc:5484] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fb13c94e664ec280fad2968e515d19 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "02fb13c94e664ec280fad2968e515d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:05.703557  9797 raft_consensus.cc:482] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:05.703681  9797 raft_consensus.cc:504] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:05.704280  9797 leader_election.cc:288] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923)
I20240501 20:09:05.704566  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d205586c09e8445f91ea63579e8b8ded" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:05.704717  8783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d205586c09e8445f91ea63579e8b8ded" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" is_pre_election: true
I20240501 20:09:05.704823  9044 raft_consensus.cc:2454] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:09:05.704918  8783 raft_consensus.cc:2454] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:09:05.705224  9241 leader_election.cc:302] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3b6c3b236a5d40bf99869152603b5524; no voters: 
I20240501 20:09:05.705462  9797 raft_consensus.cc:2781] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:05.705560  9797 raft_consensus.cc:482] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:05.705642  9797 raft_consensus.cc:3037] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.707728  9797 raft_consensus.cc:504] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:05.708230  9797 leader_election.cc:288] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923)
I20240501 20:09:05.708501  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d205586c09e8445f91ea63579e8b8ded" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:05.708698  9044 raft_consensus.cc:3037] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.708679  8783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d205586c09e8445f91ea63579e8b8ded" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
I20240501 20:09:05.708878  8783 raft_consensus.cc:3037] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.711354  9044 raft_consensus.cc:2454] T d205586c09e8445f91ea63579e8b8ded P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:09:05.711387  8783 raft_consensus.cc:2454] T d205586c09e8445f91ea63579e8b8ded P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:09:05.711701  9241 leader_election.cc:302] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3b6c3b236a5d40bf99869152603b5524; no voters: 
I20240501 20:09:05.711951  9797 raft_consensus.cc:2781] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:05.712251  9797 raft_consensus.cc:686] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [term 1 LEADER]: Becoming Leader. State: Replica: 23543295bda14131a70bd31c573454f6, State: Running, Role: LEADER
I20240501 20:09:05.712504  9797 consensus_queue.cc:237] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:05.714656  8504 catalog_manager.cc:5484] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23543295bda14131a70bd31c573454f6 (127.8.68.198). New cstate: current_term: 1 leader_uuid: "23543295bda14131a70bd31c573454f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:05.912204  9579 raft_consensus.cc:482] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:05.912353  9579 raft_consensus.cc:504] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:05.912891  9579 leader_election.cc:288] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:05.913228  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bf7725b0b94c2eac42fca017b10a2f" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:05.913292  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bf7725b0b94c2eac42fca017b10a2f" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:05.913425  9044 raft_consensus.cc:2454] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:05.913494  9306 raft_consensus.cc:2454] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:05.913784  8717 leader_election.cc:302] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:05.914048  9579 raft_consensus.cc:2781] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:05.914145  9579 raft_consensus.cc:482] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:05.914225  9579 raft_consensus.cc:3037] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.916283  9579 raft_consensus.cc:504] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:05.916859  9579 leader_election.cc:288] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:05.917309  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bf7725b0b94c2eac42fca017b10a2f" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:05.917382  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bf7725b0b94c2eac42fca017b10a2f" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:05.917547  9306 raft_consensus.cc:3037] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.917541  9044 raft_consensus.cc:3037] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:05.919270  9306 raft_consensus.cc:2454] T 43bf7725b0b94c2eac42fca017b10a2f P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:05.919399  9044 raft_consensus.cc:2454] T 43bf7725b0b94c2eac42fca017b10a2f P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:05.919581  8716 leader_election.cc:302] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:05.919843  9579 raft_consensus.cc:2781] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:05.920141  9579 raft_consensus.cc:686] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:09:05.920413  9579 consensus_queue.cc:237] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:05.923082  8504 catalog_manager.cc:5484] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:06.102196  9790 consensus_queue.cc:1035] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:06.106803  9887 consensus_queue.cc:1035] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:06.203022  9797 consensus_queue.cc:1035] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:06.215755  9889 consensus_queue.cc:1035] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:06.348702  9893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:06.349030  9893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:06.357908  9893 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:06.361994  9579 consensus_queue.cc:1035] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:06.366609  9579 consensus_queue.cc:1035] T 43bf7725b0b94c2eac42fca017b10a2f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'

INSERT report
    rows total: 2000
    time total: 346.636 ms
  time per row: 0.173318 ms
I20240501 20:09:06.862967  8501 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:06.872888  8631 tablet_service.cc:1467] Processing CreateTablet for tablet fee23b1bdb8249b68a8464928344277f (DEFAULT_TABLE table=rebalance_test_table_6 [id=f9e21362ec834c4ab5a4dfba130d63c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.873368  8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee23b1bdb8249b68a8464928344277f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.873454  9025 tablet_service.cc:1467] Processing CreateTablet for tablet a013dd2c5905470282f770673d0c1cef (DEFAULT_TABLE table=rebalance_test_table_6 [id=f9e21362ec834c4ab5a4dfba130d63c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.873713  8893 tablet_service.cc:1467] Processing CreateTablet for tablet fee23b1bdb8249b68a8464928344277f (DEFAULT_TABLE table=rebalance_test_table_6 [id=f9e21362ec834c4ab5a4dfba130d63c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.873842  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a013dd2c5905470282f770673d0c1cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.874080  8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee23b1bdb8249b68a8464928344277f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.874985  8632 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1bca656e234c68b8cfd0b1f2523f86 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f9e21362ec834c4ab5a4dfba130d63c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.875341  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1bca656e234c68b8cfd0b1f2523f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.878752  9156 tablet_service.cc:1467] Processing CreateTablet for tablet fee23b1bdb8249b68a8464928344277f (DEFAULT_TABLE table=rebalance_test_table_6 [id=f9e21362ec834c4ab5a4dfba130d63c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.879175  9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee23b1bdb8249b68a8464928344277f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.883412  8762 tablet_service.cc:1467] Processing CreateTablet for tablet a013dd2c5905470282f770673d0c1cef (DEFAULT_TABLE table=rebalance_test_table_6 [id=f9e21362ec834c4ab5a4dfba130d63c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.883719  9286 tablet_service.cc:1467] Processing CreateTablet for tablet a013dd2c5905470282f770673d0c1cef (DEFAULT_TABLE table=rebalance_test_table_6 [id=f9e21362ec834c4ab5a4dfba130d63c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.883874  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a013dd2c5905470282f770673d0c1cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.884126  9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a013dd2c5905470282f770673d0c1cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.884706  9966 tablet_bootstrap.cc:492] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:06.886234  9966 tablet_bootstrap.cc:654] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.887223  9155 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1bca656e234c68b8cfd0b1f2523f86 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f9e21362ec834c4ab5a4dfba130d63c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.887609  9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1bca656e234c68b8cfd0b1f2523f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.890717  8894 tablet_service.cc:1467] Processing CreateTablet for tablet 7f1bca656e234c68b8cfd0b1f2523f86 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f9e21362ec834c4ab5a4dfba130d63c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.891095  8894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f1bca656e234c68b8cfd0b1f2523f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.900750  9968 tablet_bootstrap.cc:492] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:06.902998  9964 tablet_bootstrap.cc:492] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:06.904170  9966 tablet_bootstrap.cc:492] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:06.904405  9966 ts_tablet_manager.cc:1367] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.000s
I20240501 20:09:06.905138  9964 tablet_bootstrap.cc:654] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.905215  9966 raft_consensus.cc:348] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:06.905509  9966 raft_consensus.cc:374] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.905645  9966 raft_consensus.cc:729] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.905980  9966 consensus_queue.cc:260] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:06.907506  9968 tablet_bootstrap.cc:654] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.910318  9970 tablet_bootstrap.cc:492] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:06.912351  9970 tablet_bootstrap.cc:654] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.913359  9969 tablet_bootstrap.cc:492] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:06.914613  9968 tablet_bootstrap.cc:492] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:06.914798  9968 ts_tablet_manager.cc:1367] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.002s
I20240501 20:09:06.915124  9964 tablet_bootstrap.cc:492] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:06.915267  9964 ts_tablet_manager.cc:1367] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240501 20:09:06.915572  9968 raft_consensus.cc:348] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:06.915764  9968 raft_consensus.cc:374] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.915838  9968 raft_consensus.cc:729] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.916002  9964 raft_consensus.cc:348] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:06.916191  9964 raft_consensus.cc:374] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.916076  9968 consensus_queue.cc:260] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:06.916265  9964 raft_consensus.cc:729] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.916551  9969 tablet_bootstrap.cc:654] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.916464  9964 consensus_queue.cc:260] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:06.917014  9964 ts_tablet_manager.cc:1398] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:09:06.917282  9964 tablet_bootstrap.cc:492] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:06.918745  9964 tablet_bootstrap.cc:654] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.921360  9965 tablet_bootstrap.cc:492] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:06.923513  9968 ts_tablet_manager.cc:1398] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.001s
I20240501 20:09:06.923520  9966 ts_tablet_manager.cc:1398] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.019s	user 0.010s	sys 0.005s
I20240501 20:09:06.924815  9965 tablet_bootstrap.cc:654] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.925742  9966 tablet_bootstrap.cc:492] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:06.926203  9969 tablet_bootstrap.cc:492] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:06.926417  9969 ts_tablet_manager.cc:1367] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.013s	user 0.006s	sys 0.000s
I20240501 20:09:06.927201  9969 raft_consensus.cc:348] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:06.927484  9966 tablet_bootstrap.cc:654] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.928440  9969 raft_consensus.cc:374] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.928558  9969 raft_consensus.cc:729] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.928627  9970 tablet_bootstrap.cc:492] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:06.928927  9970 ts_tablet_manager.cc:1367] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.002s
I20240501 20:09:06.928839  9969 consensus_queue.cc:260] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:06.929570  9964 tablet_bootstrap.cc:492] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:06.929592  9969 ts_tablet_manager.cc:1398] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:06.929751  9964 ts_tablet_manager.cc:1367] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.013s	user 0.007s	sys 0.000s
I20240501 20:09:06.930521  9964 raft_consensus.cc:348] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:06.930858  9964 raft_consensus.cc:374] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.930963  9964 raft_consensus.cc:729] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.931231  9964 consensus_queue.cc:260] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:06.932205  9964 ts_tablet_manager.cc:1398] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:06.932727  9966 tablet_bootstrap.cc:492] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:06.932924  9966 ts_tablet_manager.cc:1367] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.004s
I20240501 20:09:06.931391  9970 raft_consensus.cc:348] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:06.933775  9970 raft_consensus.cc:374] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.933921  9970 raft_consensus.cc:729] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.934152  9970 consensus_queue.cc:260] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:06.934746  9966 raft_consensus.cc:348] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:06.935029  9966 raft_consensus.cc:374] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.935146  9966 raft_consensus.cc:729] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.935213  9970 ts_tablet_manager.cc:1398] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240501 20:09:06.935411  9966 consensus_queue.cc:260] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:06.935758  9970 tablet_bootstrap.cc:492] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:06.936024  9966 ts_tablet_manager.cc:1398] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 20:09:06.937319  9970 tablet_bootstrap.cc:654] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.939749  9970 tablet_bootstrap.cc:492] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:06.939976  9970 ts_tablet_manager.cc:1367] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240501 20:09:06.940836  9965 tablet_bootstrap.cc:492] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:06.940730  9970 raft_consensus.cc:348] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:06.940995  9965 ts_tablet_manager.cc:1367] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.007s
I20240501 20:09:06.941119  9970 raft_consensus.cc:374] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.941221  9970 raft_consensus.cc:729] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.941465  9970 consensus_queue.cc:260] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:06.941689  9965 raft_consensus.cc:348] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:06.941907  9965 raft_consensus.cc:374] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.942001  9965 raft_consensus.cc:729] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.942005  9970 ts_tablet_manager.cc:1398] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:06.942224  9965 consensus_queue.cc:260] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:06.942720  9965 ts_tablet_manager.cc:1398] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:07.038625  9896 raft_consensus.cc:482] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:07.038789  9896 raft_consensus.cc:504] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:07.039407  9896 leader_election.cc:288] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:07.039790  9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1bca656e234c68b8cfd0b1f2523f86" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:07.039896  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1bca656e234c68b8cfd0b1f2523f86" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:07.040040  9175 raft_consensus.cc:2454] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:07.040113  8652 raft_consensus.cc:2454] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:07.040482  8849 leader_election.cc:302] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:07.040733  9896 raft_consensus.cc:2781] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:07.040829  9896 raft_consensus.cc:482] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:07.040906  9896 raft_consensus.cc:3037] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.042737  9896 raft_consensus.cc:504] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:07.043203  9896 leader_election.cc:288] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Requested vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:07.043560  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1bca656e234c68b8cfd0b1f2523f86" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:07.043567  9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f1bca656e234c68b8cfd0b1f2523f86" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:07.043725  9175 raft_consensus.cc:3037] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.043731  8652 raft_consensus.cc:3037] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.045431  9175 raft_consensus.cc:2454] T 7f1bca656e234c68b8cfd0b1f2523f86 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:07.045475  8652 raft_consensus.cc:2454] T 7f1bca656e234c68b8cfd0b1f2523f86 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:07.045787  8848 leader_election.cc:302] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:07.046029  9896 raft_consensus.cc:2781] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:07.046182  9896 raft_consensus.cc:686] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Running, Role: LEADER
I20240501 20:09:07.046468  9896 consensus_queue.cc:237] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:07.048825  8501 catalog_manager.cc:5484] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fb13c94e664ec280fad2968e515d19 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "02fb13c94e664ec280fad2968e515d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:07.109504  9896 raft_consensus.cc:482] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:07.109651  9896 raft_consensus.cc:504] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:07.110128  9896 leader_election.cc:288] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:07.110515  9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee23b1bdb8249b68a8464928344277f" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:07.110531  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee23b1bdb8249b68a8464928344277f" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:07.110698  8652 raft_consensus.cc:2454] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:07.110693  9175 raft_consensus.cc:2454] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:07.111018  8849 leader_election.cc:302] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:07.111253  9896 raft_consensus.cc:2781] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:07.111328  9896 raft_consensus.cc:482] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:07.111423  9896 raft_consensus.cc:3037] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.113109  9896 raft_consensus.cc:504] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:07.113591  9896 leader_election.cc:288] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Requested vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:07.113951  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee23b1bdb8249b68a8464928344277f" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:07.113979  9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee23b1bdb8249b68a8464928344277f" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:07.114128  8652 raft_consensus.cc:3037] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.114145  9175 raft_consensus.cc:3037] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.116734  9175 raft_consensus.cc:2454] T fee23b1bdb8249b68a8464928344277f P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:07.116745  8652 raft_consensus.cc:2454] T fee23b1bdb8249b68a8464928344277f P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:07.117065  8848 leader_election.cc:302] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:07.117297  9896 raft_consensus.cc:2781] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:07.117435  9896 raft_consensus.cc:686] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Running, Role: LEADER
I20240501 20:09:07.117689  9896 consensus_queue.cc:237] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:07.120172  8501 catalog_manager.cc:5484] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fb13c94e664ec280fad2968e515d19 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "02fb13c94e664ec280fad2968e515d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:07.145221  9891 raft_consensus.cc:482] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:07.145368  9891 raft_consensus.cc:504] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:07.146095  9891 leader_election.cc:288] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:07.146493  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a013dd2c5905470282f770673d0c1cef" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:07.146503  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a013dd2c5905470282f770673d0c1cef" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:07.146746  9044 raft_consensus.cc:2454] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:07.146747  9306 raft_consensus.cc:2454] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:07.147236  8716 leader_election.cc:302] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:07.147526  9891 raft_consensus.cc:2781] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:07.147624  9891 raft_consensus.cc:482] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:07.147711  9891 raft_consensus.cc:3037] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.149487  9891 raft_consensus.cc:504] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:07.150054  9891 leader_election.cc:288] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:07.150434  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a013dd2c5905470282f770673d0c1cef" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:07.150521  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a013dd2c5905470282f770673d0c1cef" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:07.150605  9306 raft_consensus.cc:3037] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.150774  9044 raft_consensus.cc:3037] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.152377  9306 raft_consensus.cc:2454] T a013dd2c5905470282f770673d0c1cef P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:07.152701  9044 raft_consensus.cc:2454] T a013dd2c5905470282f770673d0c1cef P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:07.152724  8716 leader_election.cc:302] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:07.153007  9891 raft_consensus.cc:2781] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:07.153172  9891 raft_consensus.cc:686] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:09:07.153470  9891 consensus_queue.cc:237] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:07.155609  8501 catalog_manager.cc:5484] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:07.519389  9983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:07.519683  9983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:07.527022  9983 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 20:09:07.602679  9896 consensus_queue.cc:1035] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.617527 10010 consensus_queue.cc:1035] T 7f1bca656e234c68b8cfd0b1f2523f86 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.619168  9887 consensus_queue.cc:1035] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.627748  9891 consensus_queue.cc:1035] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.631553  9896 consensus_queue.cc:1035] T fee23b1bdb8249b68a8464928344277f P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.691318  9891 consensus_queue.cc:1035] T a013dd2c5905470282f770673d0c1cef P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 307.447 ms
  time per row: 0.153724 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:09:08.842399  8504 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:08.850786  8762 tablet_service.cc:1467] Processing CreateTablet for tablet 009d7f2fece0434795136dda912beb51 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1dc7e72c86df447987e3d0327190fce2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.851004  9286 tablet_service.cc:1467] Processing CreateTablet for tablet 009d7f2fece0434795136dda912beb51 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1dc7e72c86df447987e3d0327190fce2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.851239  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009d7f2fece0434795136dda912beb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.851418  9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009d7f2fece0434795136dda912beb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.852000  9287 tablet_service.cc:1467] Processing CreateTablet for tablet ed8378c3ec35479ea1821e027c37866f (DEFAULT_TABLE table=rebalance_test_table_7 [id=1dc7e72c86df447987e3d0327190fce2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.852067  8894 tablet_service.cc:1467] Processing CreateTablet for tablet e3fa355226244b9ba3c1482c479ab94b (DEFAULT_TABLE table=rebalance_test_table_7 [id=1dc7e72c86df447987e3d0327190fce2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.852317  9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8378c3ec35479ea1821e027c37866f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.852442  8894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3fa355226244b9ba3c1482c479ab94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.856827  9155 tablet_service.cc:1467] Processing CreateTablet for tablet e3fa355226244b9ba3c1482c479ab94b (DEFAULT_TABLE table=rebalance_test_table_7 [id=1dc7e72c86df447987e3d0327190fce2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.857205  9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3fa355226244b9ba3c1482c479ab94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.863199  9025 tablet_service.cc:1467] Processing CreateTablet for tablet 009d7f2fece0434795136dda912beb51 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1dc7e72c86df447987e3d0327190fce2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.863605  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009d7f2fece0434795136dda912beb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.866040 10062 tablet_bootstrap.cc:492] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:08.867462 10062 tablet_bootstrap.cc:654] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.867620 10063 tablet_bootstrap.cc:492] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:08.868049 10064 tablet_bootstrap.cc:492] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:08.869035 10063 tablet_bootstrap.cc:654] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.869755 10064 tablet_bootstrap.cc:654] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.870671  9024 tablet_service.cc:1467] Processing CreateTablet for tablet ed8378c3ec35479ea1821e027c37866f (DEFAULT_TABLE table=rebalance_test_table_7 [id=1dc7e72c86df447987e3d0327190fce2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.871016  9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8378c3ec35479ea1821e027c37866f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.875649  8631 tablet_service.cc:1467] Processing CreateTablet for tablet e3fa355226244b9ba3c1482c479ab94b (DEFAULT_TABLE table=rebalance_test_table_7 [id=1dc7e72c86df447987e3d0327190fce2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.875813  8762 tablet_service.cc:1467] Processing CreateTablet for tablet ed8378c3ec35479ea1821e027c37866f (DEFAULT_TABLE table=rebalance_test_table_7 [id=1dc7e72c86df447987e3d0327190fce2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:08.876070  8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3fa355226244b9ba3c1482c479ab94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.876178  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8378c3ec35479ea1821e027c37866f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:08.882217 10068 tablet_bootstrap.cc:492] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:08.884265 10068 tablet_bootstrap.cc:654] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.884445 10064 tablet_bootstrap.cc:492] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:08.884637 10064 ts_tablet_manager.cc:1367] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.017s	user 0.003s	sys 0.003s
I20240501 20:09:08.885439 10064 raft_consensus.cc:348] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:08.885807 10064 raft_consensus.cc:374] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.885936 10064 raft_consensus.cc:729] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.886245 10064 consensus_queue.cc:260] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:08.886919 10064 ts_tablet_manager.cc:1398] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:08.888337 10063 tablet_bootstrap.cc:492] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:08.888540 10063 ts_tablet_manager.cc:1367] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.011s
I20240501 20:09:08.889926 10063 raft_consensus.cc:348] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:08.890173 10063 raft_consensus.cc:374] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.890337 10063 raft_consensus.cc:729] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.890641 10063 consensus_queue.cc:260] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:08.891314 10063 ts_tablet_manager.cc:1398] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:08.892525 10068 tablet_bootstrap.cc:492] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:08.892719 10068 ts_tablet_manager.cc:1367] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240501 20:09:08.894016 10068 raft_consensus.cc:348] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:08.894519 10068 raft_consensus.cc:374] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.894656 10068 raft_consensus.cc:729] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.894942 10068 consensus_queue.cc:260] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:08.899533 10063 tablet_bootstrap.cc:492] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:08.899911 10072 tablet_bootstrap.cc:492] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:08.901104 10063 tablet_bootstrap.cc:654] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.902580 10068 ts_tablet_manager.cc:1398] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20240501 20:09:08.902940 10072 tablet_bootstrap.cc:654] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.906482 10069 tablet_bootstrap.cc:492] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:08.907955 10063 tablet_bootstrap.cc:492] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:08.908494 10063 ts_tablet_manager.cc:1367] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.007s
I20240501 20:09:08.908592 10069 tablet_bootstrap.cc:654] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.909196 10063 raft_consensus.cc:348] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:08.909458 10063 raft_consensus.cc:374] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.909577 10063 raft_consensus.cc:729] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.909873 10063 consensus_queue.cc:260] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:08.910368 10063 ts_tablet_manager.cc:1398] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:08.910955 10072 tablet_bootstrap.cc:492] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:08.911741 10062 tablet_bootstrap.cc:492] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:08.911921 10062 ts_tablet_manager.cc:1367] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.046s	user 0.018s	sys 0.018s
I20240501 20:09:08.912621 10062 raft_consensus.cc:348] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:08.912815 10062 raft_consensus.cc:374] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.912953 10062 raft_consensus.cc:729] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.913203 10062 consensus_queue.cc:260] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:08.915964 10062 ts_tablet_manager.cc:1398] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240501 20:09:08.916774 10062 tablet_bootstrap.cc:492] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:08.918187 10062 tablet_bootstrap.cc:654] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.918534 10072 ts_tablet_manager.cc:1367] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.019s	user 0.005s	sys 0.000s
I20240501 20:09:08.919189 10072 raft_consensus.cc:348] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:08.919384 10072 raft_consensus.cc:374] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.920598 10072 raft_consensus.cc:729] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.920847 10072 consensus_queue.cc:260] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:08.921000 10069 tablet_bootstrap.cc:492] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:08.921123 10069 ts_tablet_manager.cc:1367] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.006s
I20240501 20:09:08.921736 10062 tablet_bootstrap.cc:492] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:08.922060 10062 ts_tablet_manager.cc:1367] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.001s
I20240501 20:09:08.922048 10072 ts_tablet_manager.cc:1398] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:08.922531 10069 raft_consensus.cc:348] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:08.922708 10069 raft_consensus.cc:374] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.922819 10069 raft_consensus.cc:729] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.922791 10062 raft_consensus.cc:348] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:08.923036 10062 raft_consensus.cc:374] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.923043 10069 consensus_queue.cc:260] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:08.923163 10062 raft_consensus.cc:729] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.923413 10062 consensus_queue.cc:260] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:08.923919 10069 ts_tablet_manager.cc:1398] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 20:09:08.924216 10069 tablet_bootstrap.cc:492] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:08.924690 10062 ts_tablet_manager.cc:1398] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:08.925849 10069 tablet_bootstrap.cc:654] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:08.929088 10069 tablet_bootstrap.cc:492] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:08.929226 10069 ts_tablet_manager.cc:1367] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240501 20:09:08.929888 10069 raft_consensus.cc:348] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:08.930034 10069 raft_consensus.cc:374] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:08.930100 10069 raft_consensus.cc:729] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:08.930318 10069 consensus_queue.cc:260] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:08.930733 10069 ts_tablet_manager.cc:1398] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I20240501 20:09:09.049634 10040 raft_consensus.cc:482] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:09.049763 10040 raft_consensus.cc:504] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:09.050310 10040 leader_election.cc:288] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:09.050631  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8378c3ec35479ea1821e027c37866f" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:09.050817  9044 raft_consensus.cc:2454] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:09.050838  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8378c3ec35479ea1821e027c37866f" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:09.051033  9306 raft_consensus.cc:2454] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:09.051179  8717 leader_election.cc:302] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:09.051452 10040 raft_consensus.cc:2781] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:09.051539 10040 raft_consensus.cc:482] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:09.051616 10040 raft_consensus.cc:3037] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.053339 10040 raft_consensus.cc:504] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:09.053805 10040 leader_election.cc:288] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:09.054210  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8378c3ec35479ea1821e027c37866f" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:09.054266  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8378c3ec35479ea1821e027c37866f" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:09.054423  9306 raft_consensus.cc:3037] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.054431  9044 raft_consensus.cc:3037] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.057051  9044 raft_consensus.cc:2454] T ed8378c3ec35479ea1821e027c37866f P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:09.057051  9306 raft_consensus.cc:2454] T ed8378c3ec35479ea1821e027c37866f P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:09.057402  8717 leader_election.cc:302] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:09.057665 10040 raft_consensus.cc:2781] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:09.058027 10040 raft_consensus.cc:686] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:09:09.058310 10040 consensus_queue.cc:237] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } }
I20240501 20:09:09.061098  8509 catalog_manager.cc:5484] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:09.176255 10071 raft_consensus.cc:482] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:09.176398 10071 raft_consensus.cc:504] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:09.176982 10071 leader_election.cc:288] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:09.177423  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fa355226244b9ba3c1482c479ab94b" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:09:09.177424  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fa355226244b9ba3c1482c479ab94b" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:09.177665  8913 raft_consensus.cc:2454] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 0.
I20240501 20:09:09.177661  9176 raft_consensus.cc:2454] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 0.
I20240501 20:09:09.178086  8585 leader_election.cc:302] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:09.178365 10071 raft_consensus.cc:2781] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:09.178488 10071 raft_consensus.cc:482] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:09.178578 10071 raft_consensus.cc:3037] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.180536 10071 raft_consensus.cc:504] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:09.181068 10071 leader_election.cc:288] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:09.181418  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fa355226244b9ba3c1482c479ab94b" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19"
I20240501 20:09:09.181474  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fa355226244b9ba3c1482c479ab94b" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:09.181636  8913 raft_consensus.cc:3037] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.181649  9176 raft_consensus.cc:3037] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.184257  9176 raft_consensus.cc:2454] T e3fa355226244b9ba3c1482c479ab94b P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 1.
I20240501 20:09:09.184265  8913 raft_consensus.cc:2454] T e3fa355226244b9ba3c1482c479ab94b P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 1.
I20240501 20:09:09.184635  8587 leader_election.cc:302] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea5761ef47546768883016105c52225, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:09.184913 10071 raft_consensus.cc:2781] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:09.185274 10071 raft_consensus.cc:686] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 LEADER]: Becoming Leader. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Running, Role: LEADER
I20240501 20:09:09.185756 10071 consensus_queue.cc:237] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:09.189086  8509 catalog_manager.cc:5484] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:09.209889  9797 raft_consensus.cc:482] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:09.210019  9797 raft_consensus.cc:504] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:09.210660  9797 leader_election.cc:288] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:09.211055  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009d7f2fece0434795136dda912beb51" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:09.211125  8783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009d7f2fece0434795136dda912beb51" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" is_pre_election: true
I20240501 20:09:09.211251  9044 raft_consensus.cc:2454] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:09:09.211333  8783 raft_consensus.cc:2454] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 0.
I20240501 20:09:09.211576  9241 leader_election.cc:302] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3b6c3b236a5d40bf99869152603b5524; no voters: 
I20240501 20:09:09.211796  9797 raft_consensus.cc:2781] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:09.211905  9797 raft_consensus.cc:482] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:09.211989  9797 raft_consensus.cc:3037] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.213670  9797 raft_consensus.cc:504] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:09.214144  9797 leader_election.cc:288] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:09.214570  8783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009d7f2fece0434795136dda912beb51" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
I20240501 20:09:09.214620  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009d7f2fece0434795136dda912beb51" candidate_uuid: "23543295bda14131a70bd31c573454f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:09.214747  8783 raft_consensus.cc:3037] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.214782  9044 raft_consensus.cc:3037] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.217342  9044 raft_consensus.cc:2454] T 009d7f2fece0434795136dda912beb51 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:09:09.217352  8783 raft_consensus.cc:2454] T 009d7f2fece0434795136dda912beb51 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23543295bda14131a70bd31c573454f6 in term 1.
I20240501 20:09:09.217691  9240 leader_election.cc:302] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:09.217962  9797 raft_consensus.cc:2781] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:09.218266  9797 raft_consensus.cc:686] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [term 1 LEADER]: Becoming Leader. State: Replica: 23543295bda14131a70bd31c573454f6, State: Running, Role: LEADER
I20240501 20:09:09.218554  9797 consensus_queue.cc:237] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:09.221236  8509 catalog_manager.cc:5484] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23543295bda14131a70bd31c573454f6 (127.8.68.198). New cstate: current_term: 1 leader_uuid: "23543295bda14131a70bd31c573454f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:09.529897 10079 consensus_queue.cc:1035] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:09.544324 10079 consensus_queue.cc:1035] T ed8378c3ec35479ea1821e027c37866f P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:09.631049 10085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:09.631392 10085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:09.639333 10085 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 20:09:09.687338 10071 consensus_queue.cc:1035] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:09.700637 10071 consensus_queue.cc:1035] T e3fa355226244b9ba3c1482c479ab94b P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:09.745316 10083 consensus_queue.cc:1035] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:09.751621 10083 consensus_queue.cc:1035] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 295.23 ms
  time per row: 0.147615 ms
I20240501 20:09:10.092336  8509 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:10.100752  8894 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee63f84c8ff4bdd86c2302163f25e6c (DEFAULT_TABLE table=rebalance_test_table_8 [id=936f37837cf34947a0c699015f6c752f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.100979  8631 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee63f84c8ff4bdd86c2302163f25e6c (DEFAULT_TABLE table=rebalance_test_table_8 [id=936f37837cf34947a0c699015f6c752f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.101209  8894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee63f84c8ff4bdd86c2302163f25e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.101154  9155 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee63f84c8ff4bdd86c2302163f25e6c (DEFAULT_TABLE table=rebalance_test_table_8 [id=936f37837cf34947a0c699015f6c752f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.101367  8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee63f84c8ff4bdd86c2302163f25e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.101558  9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee63f84c8ff4bdd86c2302163f25e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.102497  9025 tablet_service.cc:1467] Processing CreateTablet for tablet c8b7690f910048ab9d7ca0e9c8c57006 (DEFAULT_TABLE table=rebalance_test_table_8 [id=936f37837cf34947a0c699015f6c752f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.102880  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b7690f910048ab9d7ca0e9c8c57006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.109290  8893 tablet_service.cc:1467] Processing CreateTablet for tablet 122a2e00098f4ca9b91509a85c6c5860 (DEFAULT_TABLE table=rebalance_test_table_8 [id=936f37837cf34947a0c699015f6c752f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.109642  8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122a2e00098f4ca9b91509a85c6c5860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.109624  8632 tablet_service.cc:1467] Processing CreateTablet for tablet 122a2e00098f4ca9b91509a85c6c5860 (DEFAULT_TABLE table=rebalance_test_table_8 [id=936f37837cf34947a0c699015f6c752f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.109969  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122a2e00098f4ca9b91509a85c6c5860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.114548 10154 tablet_bootstrap.cc:492] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:10.114650  8762 tablet_service.cc:1467] Processing CreateTablet for tablet c8b7690f910048ab9d7ca0e9c8c57006 (DEFAULT_TABLE table=rebalance_test_table_8 [id=936f37837cf34947a0c699015f6c752f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.115028  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b7690f910048ab9d7ca0e9c8c57006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.116078 10154 tablet_bootstrap.cc:654] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.116726 10152 tablet_bootstrap.cc:492] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:10.117588 10153 tablet_bootstrap.cc:492] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:10.118534 10152 tablet_bootstrap.cc:654] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.119122 10153 tablet_bootstrap.cc:654] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.119057  9287 tablet_service.cc:1467] Processing CreateTablet for tablet c8b7690f910048ab9d7ca0e9c8c57006 (DEFAULT_TABLE table=rebalance_test_table_8 [id=936f37837cf34947a0c699015f6c752f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.119463  9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b7690f910048ab9d7ca0e9c8c57006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.121565 10155 tablet_bootstrap.cc:492] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:10.123003  9156 tablet_service.cc:1467] Processing CreateTablet for tablet 122a2e00098f4ca9b91509a85c6c5860 (DEFAULT_TABLE table=rebalance_test_table_8 [id=936f37837cf34947a0c699015f6c752f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:10.123112 10155 tablet_bootstrap.cc:654] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.123350  9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122a2e00098f4ca9b91509a85c6c5860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:10.125425 10153 tablet_bootstrap.cc:492] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:10.125558 10153 ts_tablet_manager.cc:1367] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240501 20:09:10.125747 10155 tablet_bootstrap.cc:492] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:10.125859 10155 ts_tablet_manager.cc:1367] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 20:09:10.126336 10153 raft_consensus.cc:348] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:10.126535 10153 raft_consensus.cc:374] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.126610 10153 raft_consensus.cc:729] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.126538 10155 raft_consensus.cc:348] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:10.126734 10155 raft_consensus.cc:374] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.126829 10155 raft_consensus.cc:729] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.127116 10155 consensus_queue.cc:260] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:10.127676 10155 ts_tablet_manager.cc:1398] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:10.128803 10153 consensus_queue.cc:260] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:10.129912 10153 ts_tablet_manager.cc:1398] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:09:10.133091 10153 tablet_bootstrap.cc:492] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:10.136152 10160 tablet_bootstrap.cc:492] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:10.137178 10154 tablet_bootstrap.cc:492] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:10.137342 10154 ts_tablet_manager.cc:1367] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.023s	user 0.000s	sys 0.009s
I20240501 20:09:10.138105 10154 raft_consensus.cc:348] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:10.138332 10154 raft_consensus.cc:374] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.138438 10154 raft_consensus.cc:729] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.138686 10160 tablet_bootstrap.cc:654] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.138713 10154 consensus_queue.cc:260] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:10.139262 10154 ts_tablet_manager.cc:1398] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:10.139755 10153 tablet_bootstrap.cc:654] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.140036 10154 tablet_bootstrap.cc:492] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:10.140923 10161 tablet_bootstrap.cc:492] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:10.141465 10154 tablet_bootstrap.cc:654] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.142616 10161 tablet_bootstrap.cc:654] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.142683 10152 tablet_bootstrap.cc:492] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:10.142841 10152 ts_tablet_manager.cc:1367] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.004s
I20240501 20:09:10.143586 10152 raft_consensus.cc:348] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:10.143785 10152 raft_consensus.cc:374] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.143857 10152 raft_consensus.cc:729] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.144050 10152 consensus_queue.cc:260] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:10.144554 10152 ts_tablet_manager.cc:1398] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:09:10.144802 10152 tablet_bootstrap.cc:492] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:10.146198 10152 tablet_bootstrap.cc:654] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:10.149031 10161 tablet_bootstrap.cc:492] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:10.149684 10152 tablet_bootstrap.cc:492] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:10.149829 10152 ts_tablet_manager.cc:1367] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 20:09:10.149627 10161 ts_tablet_manager.cc:1367] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240501 20:09:10.150521 10152 raft_consensus.cc:348] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:10.150821 10152 raft_consensus.cc:374] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.150943 10152 raft_consensus.cc:729] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.151235 10152 consensus_queue.cc:260] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:10.151361 10161 raft_consensus.cc:348] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:10.151715 10154 tablet_bootstrap.cc:492] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:10.152004 10154 ts_tablet_manager.cc:1367] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.012s	user 0.009s	sys 0.000s
I20240501 20:09:10.152261 10160 tablet_bootstrap.cc:492] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:10.152465 10160 ts_tablet_manager.cc:1367] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.017s	user 0.002s	sys 0.005s
I20240501 20:09:10.152850 10154 raft_consensus.cc:348] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:10.153110 10154 raft_consensus.cc:374] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.153213 10154 raft_consensus.cc:729] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.153491 10154 consensus_queue.cc:260] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:10.153751 10160 raft_consensus.cc:348] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:10.154045 10160 raft_consensus.cc:374] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.154196 10152 ts_tablet_manager.cc:1398] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:09:10.154187 10160 raft_consensus.cc:729] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.154273 10154 ts_tablet_manager.cc:1398] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:10.154526 10160 consensus_queue.cc:260] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:10.156478 10161 raft_consensus.cc:374] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.156587 10161 raft_consensus.cc:729] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.156812 10161 consensus_queue.cc:260] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:10.157502 10161 ts_tablet_manager.cc:1398] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.000s
I20240501 20:09:10.158587 10153 tablet_bootstrap.cc:492] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:10.158733 10153 ts_tablet_manager.cc:1367] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.026s	user 0.010s	sys 0.000s
I20240501 20:09:10.159476 10153 raft_consensus.cc:348] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:10.159713 10153 raft_consensus.cc:374] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:10.159816 10153 raft_consensus.cc:729] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:10.159981 10160 ts_tablet_manager.cc:1398] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240501 20:09:10.160084 10153 consensus_queue.cc:260] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:10.160604 10153 ts_tablet_manager.cc:1398] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:10.168248 10071 raft_consensus.cc:482] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:10.168372 10071 raft_consensus.cc:504] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:10.168866 10071 leader_election.cc:288] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:10.169200  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee63f84c8ff4bdd86c2302163f25e6c" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:09:10.169294  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee63f84c8ff4bdd86c2302163f25e6c" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:10.169430  8913 raft_consensus.cc:2454] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 0.
I20240501 20:09:10.169468  9176 raft_consensus.cc:2454] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 0.
I20240501 20:09:10.169814  8585 leader_election.cc:302] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:10.170081 10071 raft_consensus.cc:2781] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:10.170161 10071 raft_consensus.cc:482] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:10.170253 10071 raft_consensus.cc:3037] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.172060 10071 raft_consensus.cc:504] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:10.172506 10071 leader_election.cc:288] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:10.172834  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee63f84c8ff4bdd86c2302163f25e6c" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19"
I20240501 20:09:10.172878  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee63f84c8ff4bdd86c2302163f25e6c" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:10.172959  8913 raft_consensus.cc:3037] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.173030  9176 raft_consensus.cc:3037] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.174683  8913 raft_consensus.cc:2454] T 3ee63f84c8ff4bdd86c2302163f25e6c P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 1.
I20240501 20:09:10.174743  9176 raft_consensus.cc:2454] T 3ee63f84c8ff4bdd86c2302163f25e6c P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 1.
I20240501 20:09:10.175014  8585 leader_election.cc:302] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:10.175282 10071 raft_consensus.cc:2781] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:10.175464 10071 raft_consensus.cc:686] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 LEADER]: Becoming Leader. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Running, Role: LEADER
I20240501 20:09:10.175710 10071 consensus_queue.cc:237] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:10.178474  8506 catalog_manager.cc:5484] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:10.200251 10071 raft_consensus.cc:482] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:10.200373 10071 raft_consensus.cc:504] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:10.200932 10071 leader_election.cc:288] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647)
I20240501 20:09:10.201105  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122a2e00098f4ca9b91509a85c6c5860" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:10.201242  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122a2e00098f4ca9b91509a85c6c5860" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:09:10.201272  9176 raft_consensus.cc:2454] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 0.
I20240501 20:09:10.201457  8913 raft_consensus.cc:2454] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 0.
I20240501 20:09:10.201629  8587 leader_election.cc:302] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea5761ef47546768883016105c52225, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:10.201902 10071 raft_consensus.cc:2781] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:10.201990 10071 raft_consensus.cc:482] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:10.202059 10071 raft_consensus.cc:3037] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.203794 10071 raft_consensus.cc:504] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:10.204391 10071 leader_election.cc:288] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647)
I20240501 20:09:10.204666  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122a2e00098f4ca9b91509a85c6c5860" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:10.204849  9176 raft_consensus.cc:3037] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.204866  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122a2e00098f4ca9b91509a85c6c5860" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19"
I20240501 20:09:10.205025  8913 raft_consensus.cc:3037] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.207484  9176 raft_consensus.cc:2454] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 1.
I20240501 20:09:10.207614  8913 raft_consensus.cc:2454] T 122a2e00098f4ca9b91509a85c6c5860 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 1.
I20240501 20:09:10.207827  8587 leader_election.cc:302] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea5761ef47546768883016105c52225, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:10.208099 10071 raft_consensus.cc:2781] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:10.208221 10071 raft_consensus.cc:686] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 LEADER]: Becoming Leader. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Running, Role: LEADER
I20240501 20:09:10.208453 10071 consensus_queue.cc:237] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:10.211158  8506 catalog_manager.cc:5484] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:10.266820 10079 raft_consensus.cc:482] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:10.266979 10079 raft_consensus.cc:504] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:10.267575 10079 leader_election.cc:288] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:10.267957  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b7690f910048ab9d7ca0e9c8c57006" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:10.268029  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b7690f910048ab9d7ca0e9c8c57006" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:10.268153  9306 raft_consensus.cc:2454] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:10.268237  9044 raft_consensus.cc:2454] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:10.268549  8716 leader_election.cc:302] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:10.268817 10079 raft_consensus.cc:2781] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:10.268916 10079 raft_consensus.cc:482] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:10.268978 10079 raft_consensus.cc:3037] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.270852 10079 raft_consensus.cc:504] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:10.271327 10079 leader_election.cc:288] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:10.271728  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b7690f910048ab9d7ca0e9c8c57006" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:10.271764  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b7690f910048ab9d7ca0e9c8c57006" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:10.271919  9044 raft_consensus.cc:3037] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.271931  9306 raft_consensus.cc:3037] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.274506  9306 raft_consensus.cc:2454] T c8b7690f910048ab9d7ca0e9c8c57006 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:10.274519  9044 raft_consensus.cc:2454] T c8b7690f910048ab9d7ca0e9c8c57006 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:10.274890  8716 leader_election.cc:302] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:10.275142 10079 raft_consensus.cc:2781] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:10.275272 10079 raft_consensus.cc:686] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:09:10.275522 10079 consensus_queue.cc:237] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:10.278234  8506 catalog_manager.cc:5484] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:10.587741 10071 consensus_queue.cc:1035] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.594524 10071 consensus_queue.cc:1035] T 3ee63f84c8ff4bdd86c2302163f25e6c P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:10.619309 10170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:10.619585 10170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:10.627308 10170 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 20:09:10.677999 10079 consensus_queue.cc:1035] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.703016 10079 consensus_queue.cc:1035] T c8b7690f910048ab9d7ca0e9c8c57006 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.809293  9175 raft_consensus.cc:1261] T 122a2e00098f4ca9b91509a85c6c5860 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Refusing update from remote peer a0e6c99bc66b433f9d5cf9340dad0cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:10.810016 10071 consensus_queue.cc:1035] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.826134 10215 consensus_queue.cc:1035] T 122a2e00098f4ca9b91509a85c6c5860 P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.864596 10214 mvcc.cc:205] Tried to move back new op lower bound from 7022977641696686080 to 7022977639256461312. Current Snapshot: MvccSnapshot[applied={T|T < 7022977641696686080}]
I20240501 20:09:10.889784 10220 mvcc.cc:205] Tried to move back new op lower bound from 7022977641696686080 to 7022977639256461312. Current Snapshot: MvccSnapshot[applied={T|T < 7022977641696686080}]
I20240501 20:09:10.896991 10175 mvcc.cc:205] Tried to move back new op lower bound from 7022977641696686080 to 7022977639256461312. Current Snapshot: MvccSnapshot[applied={T|T < 7022977641696686080}]

INSERT report
    rows total: 2000
    time total: 252.164 ms
  time per row: 0.126082 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      4       1192 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:09:11.919754  8505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:11.929667  8762 tablet_service.cc:1467] Processing CreateTablet for tablet 83c877a1b0d04d638854d654c46da343 (DEFAULT_TABLE table=rebalance_test_table_9 [id=0a1c00039dbb4364a521f861c2085f05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.929988  9286 tablet_service.cc:1467] Processing CreateTablet for tablet b82d800828b3481e9bda31ee121a6588 (DEFAULT_TABLE table=rebalance_test_table_9 [id=0a1c00039dbb4364a521f861c2085f05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.930125  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c877a1b0d04d638854d654c46da343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.930186  9024 tablet_service.cc:1467] Processing CreateTablet for tablet b82d800828b3481e9bda31ee121a6588 (DEFAULT_TABLE table=rebalance_test_table_9 [id=0a1c00039dbb4364a521f861c2085f05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.930410  9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82d800828b3481e9bda31ee121a6588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.930634  9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82d800828b3481e9bda31ee121a6588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.934613  9287 tablet_service.cc:1467] Processing CreateTablet for tablet 83c877a1b0d04d638854d654c46da343 (DEFAULT_TABLE table=rebalance_test_table_9 [id=0a1c00039dbb4364a521f861c2085f05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.934962  9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c877a1b0d04d638854d654c46da343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.938690  9025 tablet_service.cc:1467] Processing CreateTablet for tablet 83c877a1b0d04d638854d654c46da343 (DEFAULT_TABLE table=rebalance_test_table_9 [id=0a1c00039dbb4364a521f861c2085f05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.939121  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c877a1b0d04d638854d654c46da343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.943364  8763 tablet_service.cc:1467] Processing CreateTablet for tablet b82d800828b3481e9bda31ee121a6588 (DEFAULT_TABLE table=rebalance_test_table_9 [id=0a1c00039dbb4364a521f861c2085f05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.943790  8763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82d800828b3481e9bda31ee121a6588. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.946943  8632 tablet_service.cc:1467] Processing CreateTablet for tablet 66a87a4b36cc40998d8e89ad0fb3265b (DEFAULT_TABLE table=rebalance_test_table_9 [id=0a1c00039dbb4364a521f861c2085f05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.947379  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a87a4b36cc40998d8e89ad0fb3265b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.951298  9155 tablet_service.cc:1467] Processing CreateTablet for tablet 66a87a4b36cc40998d8e89ad0fb3265b (DEFAULT_TABLE table=rebalance_test_table_9 [id=0a1c00039dbb4364a521f861c2085f05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.951712  9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a87a4b36cc40998d8e89ad0fb3265b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.952159 10242 tablet_bootstrap.cc:492] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:11.954758  8893 tablet_service.cc:1467] Processing CreateTablet for tablet 66a87a4b36cc40998d8e89ad0fb3265b (DEFAULT_TABLE table=rebalance_test_table_9 [id=0a1c00039dbb4364a521f861c2085f05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:11.955008 10242 tablet_bootstrap.cc:654] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.955178  8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a87a4b36cc40998d8e89ad0fb3265b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:11.963068 10244 tablet_bootstrap.cc:492] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:11.963125 10242 tablet_bootstrap.cc:492] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:11.963268 10242 ts_tablet_manager.cc:1367] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.005s
I20240501 20:09:11.964103 10242 raft_consensus.cc:348] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:11.964432 10242 raft_consensus.cc:374] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.964568 10242 raft_consensus.cc:729] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.964694 10244 tablet_bootstrap.cc:654] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.964900 10242 consensus_queue.cc:260] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:11.965620 10242 ts_tablet_manager.cc:1398] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:11.966259 10242 tablet_bootstrap.cc:492] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:11.968451 10242 tablet_bootstrap.cc:654] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.974412 10245 tablet_bootstrap.cc:492] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:11.977244 10246 tablet_bootstrap.cc:492] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:11.977691 10249 tablet_bootstrap.cc:492] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:11.979084 10246 tablet_bootstrap.cc:654] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.979207 10245 tablet_bootstrap.cc:654] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.981462 10241 tablet_bootstrap.cc:492] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:11.982631 10249 tablet_bootstrap.cc:654] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.984208 10241 tablet_bootstrap.cc:654] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.986150 10249 tablet_bootstrap.cc:492] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:11.986342 10249 ts_tablet_manager.cc:1367] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.004s
I20240501 20:09:11.987079 10249 raft_consensus.cc:348] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:11.987579 10249 raft_consensus.cc:374] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.987828 10249 raft_consensus.cc:729] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.988224 10249 consensus_queue.cc:260] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:11.988916 10244 tablet_bootstrap.cc:492] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:11.989111 10244 ts_tablet_manager.cc:1367] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.026s	user 0.007s	sys 0.008s
I20240501 20:09:11.989531 10249 ts_tablet_manager.cc:1398] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:11.989938 10246 tablet_bootstrap.cc:492] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:11.990118 10246 ts_tablet_manager.cc:1367] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.013s	user 0.007s	sys 0.000s
I20240501 20:09:11.990350 10245 tablet_bootstrap.cc:492] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:11.990531 10245 ts_tablet_manager.cc:1367] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.025s	user 0.002s	sys 0.002s
I20240501 20:09:11.990866 10246 raft_consensus.cc:348] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:11.991114 10246 raft_consensus.cc:374] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.991242 10246 raft_consensus.cc:729] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.989794 10244 raft_consensus.cc:348] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:11.991313 10245 raft_consensus.cc:348] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:11.991489 10245 raft_consensus.cc:374] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.991557 10244 raft_consensus.cc:374] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.991612 10245 raft_consensus.cc:729] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.991688 10244 raft_consensus.cc:729] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.991842 10245 consensus_queue.cc:260] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:11.991866 10246 consensus_queue.cc:260] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:11.992295 10244 consensus_queue.cc:260] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:11.993543 10244 ts_tablet_manager.cc:1398] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240501 20:09:11.994098 10245 ts_tablet_manager.cc:1398] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 20:09:11.994364 10245 tablet_bootstrap.cc:492] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:11.994513 10246 ts_tablet_manager.cc:1398] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.001s
I20240501 20:09:11.995494 10241 tablet_bootstrap.cc:492] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:11.995668 10241 ts_tablet_manager.cc:1367] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.002s
I20240501 20:09:11.996402 10241 raft_consensus.cc:348] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:11.996647 10241 raft_consensus.cc:374] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:11.996788 10241 raft_consensus.cc:729] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.997071 10241 consensus_queue.cc:260] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:11.997712 10241 ts_tablet_manager.cc:1398] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:11.998005 10241 tablet_bootstrap.cc:492] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:11.998543 10245 tablet_bootstrap.cc:654] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:11.999497 10241 tablet_bootstrap.cc:654] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:12.004997 10241 tablet_bootstrap.cc:492] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:12.005199 10241 ts_tablet_manager.cc:1367] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.001s
I20240501 20:09:12.005986 10241 raft_consensus.cc:348] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:12.006268 10241 raft_consensus.cc:374] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:12.006412 10241 raft_consensus.cc:729] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:12.006701 10241 consensus_queue.cc:260] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:12.006812 10242 tablet_bootstrap.cc:492] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:12.007030 10242 ts_tablet_manager.cc:1367] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.041s	user 0.000s	sys 0.009s
I20240501 20:09:12.007288 10241 ts_tablet_manager.cc:1398] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:09:12.007862 10242 raft_consensus.cc:348] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:12.008118 10242 raft_consensus.cc:374] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:12.008194 10242 raft_consensus.cc:729] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:12.008612 10245 tablet_bootstrap.cc:492] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:12.008572 10242 consensus_queue.cc:260] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:12.008765 10245 ts_tablet_manager.cc:1367] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.000s
I20240501 20:09:12.009163 10242 ts_tablet_manager.cc:1398] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:09:12.009521 10245 raft_consensus.cc:348] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:12.009792 10245 raft_consensus.cc:374] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:12.009919 10245 raft_consensus.cc:729] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:12.010578 10245 consensus_queue.cc:260] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:12.011150 10245 ts_tablet_manager.cc:1398] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:09:12.229159 10254 raft_consensus.cc:482] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:12.229305 10254 raft_consensus.cc:504] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:12.229939 10254 leader_election.cc:288] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:12.230404  9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a87a4b36cc40998d8e89ad0fb3265b" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:12.230430  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a87a4b36cc40998d8e89ad0fb3265b" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:09:12.230631  8913 raft_consensus.cc:2454] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 0.
I20240501 20:09:12.230623  9175 raft_consensus.cc:2454] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 0.
I20240501 20:09:12.231046  8587 leader_election.cc:302] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea5761ef47546768883016105c52225, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:12.231308 10254 raft_consensus.cc:2781] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:12.231426 10254 raft_consensus.cc:482] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:12.231496 10254 raft_consensus.cc:3037] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:12.270468 10240 raft_consensus.cc:482] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:12.270617 10240 raft_consensus.cc:504] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:12.271191 10240 leader_election.cc:288] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:12.271581  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c877a1b0d04d638854d654c46da343" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:12.271764  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c877a1b0d04d638854d654c46da343" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:12.271912  9307 raft_consensus.cc:2454] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:12.272009  9044 raft_consensus.cc:2454] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:12.272361  8716 leader_election.cc:302] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:12.271965 10254 raft_consensus.cc:504] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:12.273160 10254 leader_election.cc:288] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:12.273370 10240 raft_consensus.cc:2781] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:12.273473 10240 raft_consensus.cc:482] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:12.273547 10240 raft_consensus.cc:3037] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:12.273607  9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a87a4b36cc40998d8e89ad0fb3265b" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:12.273829  9175 raft_consensus.cc:3037] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:12.276250  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a87a4b36cc40998d8e89ad0fb3265b" candidate_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19"
I20240501 20:09:12.276427  8913 raft_consensus.cc:3037] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:12.336982 10079 raft_consensus.cc:482] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:12.337170 10079 raft_consensus.cc:504] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:12.337704 10079 leader_election.cc:288] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:12.338146  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82d800828b3481e9bda31ee121a6588" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:12.338124  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82d800828b3481e9bda31ee121a6588" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:12.338388  9044 raft_consensus.cc:2454] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:12.338389  9307 raft_consensus.cc:2454] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:12.338856  8717 leader_election.cc:302] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:12.339120 10079 raft_consensus.cc:2781] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:12.339200 10079 raft_consensus.cc:482] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:12.339293 10079 raft_consensus.cc:3037] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:12.372764  9175 raft_consensus.cc:2454] T 66a87a4b36cc40998d8e89ad0fb3265b P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 1.
I20240501 20:09:12.372782  8913 raft_consensus.cc:2454] T 66a87a4b36cc40998d8e89ad0fb3265b P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e6c99bc66b433f9d5cf9340dad0cc0 in term 1.
I20240501 20:09:12.373301  8587 leader_election.cc:302] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea5761ef47546768883016105c52225, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:12.373431 10240 raft_consensus.cc:504] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:12.373688 10254 raft_consensus.cc:2781] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:12.374182 10254 raft_consensus.cc:686] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 LEADER]: Becoming Leader. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Running, Role: LEADER
I20240501 20:09:12.374513  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c877a1b0d04d638854d654c46da343" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:12.374542 10254 consensus_queue.cc:237] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:12.374751  9307 raft_consensus.cc:3037] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:12.375181 10079 raft_consensus.cc:504] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:12.375818 10079 leader_election.cc:288] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:12.376498  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82d800828b3481e9bda31ee121a6588" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:12.376681  9306 raft_consensus.cc:3037] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:12.378397  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82d800828b3481e9bda31ee121a6588" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:12.378556 10240 leader_election.cc:288] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:12.378758  9044 raft_consensus.cc:3037] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:12.379261  9045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c877a1b0d04d638854d654c46da343" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:12.379477  9045 raft_consensus.cc:3037] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:12.380093  8505 catalog_manager.cc:5484] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:12.468641  9307 raft_consensus.cc:2454] T 83c877a1b0d04d638854d654c46da343 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:12.468753  9306 raft_consensus.cc:2454] T b82d800828b3481e9bda31ee121a6588 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:12.469231  9045 raft_consensus.cc:2454] T 83c877a1b0d04d638854d654c46da343 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:12.469244  8716 leader_election.cc:302] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:12.469543  8716 leader_election.cc:302] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:12.469974 10079 raft_consensus.cc:2781] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:12.469585  9044 raft_consensus.cc:2454] T b82d800828b3481e9bda31ee121a6588 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:12.471403 10240 raft_consensus.cc:2781] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:12.473316 10079 raft_consensus.cc:686] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:09:12.473691 10240 raft_consensus.cc:686] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:09:12.473620 10079 consensus_queue.cc:237] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:12.474002 10240 consensus_queue.cc:237] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } }
I20240501 20:09:12.477705  8505 catalog_manager.cc:5484] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:12.478149  8505 catalog_manager.cc:5484] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:12.788201 10254 consensus_queue.cc:1035] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:12.806671 10254 consensus_queue.cc:1035] T 66a87a4b36cc40998d8e89ad0fb3265b P a0e6c99bc66b433f9d5cf9340dad0cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:12.825254 10262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:12.825546 10262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:12.833459 10262 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 20:09:12.866134 10240 consensus_queue.cc:1035] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:12.878557 10079 consensus_queue.cc:1035] T 83c877a1b0d04d638854d654c46da343 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:12.941970 10079 consensus_queue.cc:1035] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:12.946796 10079 consensus_queue.cc:1035] T b82d800828b3481e9bda31ee121a6588 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 202.155 ms
  time per row: 0.101077 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
     28       7248 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:09:14.085050  8505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_10"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:14.093786  8632 tablet_service.cc:1467] Processing CreateTablet for tablet 915752c40a1a4e6e88118bab3b2f3527 (DEFAULT_TABLE table=rebalance_test_table_10 [id=967b9e889e234ff485ceb6774deec742]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.093976  8893 tablet_service.cc:1467] Processing CreateTablet for tablet 915752c40a1a4e6e88118bab3b2f3527 (DEFAULT_TABLE table=rebalance_test_table_10 [id=967b9e889e234ff485ceb6774deec742]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.094262  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915752c40a1a4e6e88118bab3b2f3527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.094404  8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915752c40a1a4e6e88118bab3b2f3527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.095254  9286 tablet_service.cc:1467] Processing CreateTablet for tablet 2c66aafb88274682a73912c1f47e9b03 (DEFAULT_TABLE table=rebalance_test_table_10 [id=967b9e889e234ff485ceb6774deec742]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.095609  9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c66aafb88274682a73912c1f47e9b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.098904  8894 tablet_service.cc:1467] Processing CreateTablet for tablet 64af9fa4fb6c4f4ca992e389bf54e1ca (DEFAULT_TABLE table=rebalance_test_table_10 [id=967b9e889e234ff485ceb6774deec742]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.099257  8894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64af9fa4fb6c4f4ca992e389bf54e1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.099308  9155 tablet_service.cc:1467] Processing CreateTablet for tablet 915752c40a1a4e6e88118bab3b2f3527 (DEFAULT_TABLE table=rebalance_test_table_10 [id=967b9e889e234ff485ceb6774deec742]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.099702  9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915752c40a1a4e6e88118bab3b2f3527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.101315  8762 tablet_service.cc:1467] Processing CreateTablet for tablet 2c66aafb88274682a73912c1f47e9b03 (DEFAULT_TABLE table=rebalance_test_table_10 [id=967b9e889e234ff485ceb6774deec742]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.101688  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c66aafb88274682a73912c1f47e9b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.102653  9156 tablet_service.cc:1467] Processing CreateTablet for tablet 64af9fa4fb6c4f4ca992e389bf54e1ca (DEFAULT_TABLE table=rebalance_test_table_10 [id=967b9e889e234ff485ceb6774deec742]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.102978  9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64af9fa4fb6c4f4ca992e389bf54e1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.105003 10336 tablet_bootstrap.cc:492] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:14.105492  9024 tablet_service.cc:1467] Processing CreateTablet for tablet 2c66aafb88274682a73912c1f47e9b03 (DEFAULT_TABLE table=rebalance_test_table_10 [id=967b9e889e234ff485ceb6774deec742]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.105952  9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c66aafb88274682a73912c1f47e9b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.108776  8631 tablet_service.cc:1467] Processing CreateTablet for tablet 64af9fa4fb6c4f4ca992e389bf54e1ca (DEFAULT_TABLE table=rebalance_test_table_10 [id=967b9e889e234ff485ceb6774deec742]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:14.109160  8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64af9fa4fb6c4f4ca992e389bf54e1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:14.110066 10335 tablet_bootstrap.cc:492] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:14.111567 10335 tablet_bootstrap.cc:654] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.125241 10338 tablet_bootstrap.cc:492] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:14.126960 10337 tablet_bootstrap.cc:492] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:14.127163 10338 tablet_bootstrap.cc:654] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.128656 10336 tablet_bootstrap.cc:654] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.130052 10337 tablet_bootstrap.cc:654] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.130558 10341 tablet_bootstrap.cc:492] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:14.131281 10340 tablet_bootstrap.cc:492] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:14.132293 10341 tablet_bootstrap.cc:654] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.133046 10340 tablet_bootstrap.cc:654] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.135345 10336 tablet_bootstrap.cc:492] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:14.135349 10341 tablet_bootstrap.cc:492] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:14.135502 10341 ts_tablet_manager.cc:1367] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 20:09:14.135581 10336 ts_tablet_manager.cc:1367] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.031s	user 0.004s	sys 0.000s
I20240501 20:09:14.136296 10341 raft_consensus.cc:348] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:14.136626 10341 raft_consensus.cc:374] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.136503 10336 raft_consensus.cc:348] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:14.136755 10341 raft_consensus.cc:729] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.136991 10336 raft_consensus.cc:374] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.137006 10341 consensus_queue.cc:260] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:14.137559 10341 ts_tablet_manager.cc:1398] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:14.137143 10336 raft_consensus.cc:729] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.138523 10336 consensus_queue.cc:260] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:14.139333 10337 tablet_bootstrap.cc:492] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:14.141513 10336 ts_tablet_manager.cc:1398] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 20:09:14.142704 10336 tablet_bootstrap.cc:492] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:14.143620 10335 tablet_bootstrap.cc:492] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:14.143761 10335 ts_tablet_manager.cc:1367] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.034s	user 0.008s	sys 0.013s
I20240501 20:09:14.144454 10335 raft_consensus.cc:348] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:14.139526 10337 ts_tablet_manager.cc:1367] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.002s
I20240501 20:09:14.144655 10335 raft_consensus.cc:374] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.144742 10336 tablet_bootstrap.cc:654] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.144740 10335 raft_consensus.cc:729] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.144950 10335 consensus_queue.cc:260] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:14.145344 10337 raft_consensus.cc:348] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:14.145483 10335 ts_tablet_manager.cc:1398] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:09:14.145552 10337 raft_consensus.cc:374] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.145637 10337 raft_consensus.cc:729] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.145829 10337 consensus_queue.cc:260] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:14.146355 10337 ts_tablet_manager.cc:1398] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:09:14.146693 10337 tablet_bootstrap.cc:492] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:14.148241 10337 tablet_bootstrap.cc:654] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.150053 10338 tablet_bootstrap.cc:492] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:14.150247 10338 ts_tablet_manager.cc:1367] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.025s	user 0.004s	sys 0.018s
I20240501 20:09:14.151026 10338 raft_consensus.cc:348] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:14.151284 10338 raft_consensus.cc:374] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.151422 10338 raft_consensus.cc:729] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.151705 10338 consensus_queue.cc:260] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:14.153101 10338 ts_tablet_manager.cc:1398] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 20:09:14.155715 10337 tablet_bootstrap.cc:492] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:14.155913 10337 ts_tablet_manager.cc:1367] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.005s
I20240501 20:09:14.156634 10337 raft_consensus.cc:348] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:14.156908 10337 raft_consensus.cc:374] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.157027 10337 raft_consensus.cc:729] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.157332 10337 consensus_queue.cc:260] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:14.157572 10340 tablet_bootstrap.cc:492] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:14.157645 10336 tablet_bootstrap.cc:492] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:14.157723 10340 ts_tablet_manager.cc:1367] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.027s	user 0.008s	sys 0.012s
I20240501 20:09:14.157871 10336 ts_tablet_manager.cc:1367] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.000s
I20240501 20:09:14.157985 10337 ts_tablet_manager.cc:1398] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:09:14.158491 10340 raft_consensus.cc:348] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:14.158586 10336 raft_consensus.cc:348] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:14.158684 10340 raft_consensus.cc:374] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.158766 10336 raft_consensus.cc:374] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.158764 10340 raft_consensus.cc:729] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.158823 10336 raft_consensus.cc:729] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.158982 10340 consensus_queue.cc:260] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:14.159068 10336 consensus_queue.cc:260] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:14.159512 10340 ts_tablet_manager.cc:1398] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:14.159601 10336 ts_tablet_manager.cc:1398] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:14.160444 10340 tablet_bootstrap.cc:492] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:14.161566 10340 tablet_bootstrap.cc:654] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:14.163578 10340 tablet_bootstrap.cc:492] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:14.163709 10340 ts_tablet_manager.cc:1367] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:09:14.164340 10340 raft_consensus.cc:348] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:14.164544 10340 raft_consensus.cc:374] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:14.164623 10340 raft_consensus.cc:729] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:14.164826 10340 consensus_queue.cc:260] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:14.165288 10340 ts_tablet_manager.cc:1398] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:09:14.291386 10057 raft_consensus.cc:482] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:14.291539 10057 raft_consensus.cc:504] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:14.292320 10057 leader_election.cc:288] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647)
I20240501 20:09:14.292712  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9fa4fb6c4f4ca992e389bf54e1ca" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19" is_pre_election: true
I20240501 20:09:14.292724  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9fa4fb6c4f4ca992e389bf54e1ca" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:14.292904  8913 raft_consensus.cc:2454] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 0.
I20240501 20:09:14.292948  8652 raft_consensus.cc:2454] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 0.
I20240501 20:09:14.293298  9109 leader_election.cc:302] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:14.293534 10057 raft_consensus.cc:2781] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:14.293642 10057 raft_consensus.cc:482] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:14.293711 10057 raft_consensus.cc:3037] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.294077 10266 raft_consensus.cc:482] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:14.294219 10266 raft_consensus.cc:504] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:14.295146 10266 leader_election.cc:288] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:14.295814  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915752c40a1a4e6e88118bab3b2f3527" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:14.296022  8652 raft_consensus.cc:2454] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:14.296023  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915752c40a1a4e6e88118bab3b2f3527" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:14.296221  9176 raft_consensus.cc:2454] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:14.296566  8849 leader_election.cc:302] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:14.296852 10350 raft_consensus.cc:2781] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:14.296985 10350 raft_consensus.cc:482] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:14.297143 10350 raft_consensus.cc:3037] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.328130 10057 raft_consensus.cc:504] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:14.328130 10350 raft_consensus.cc:504] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:14.328758 10057 leader_election.cc:288] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 election: Requested vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647)
I20240501 20:09:14.328895 10350 leader_election.cc:288] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Requested vote from peers a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491), 4ea5761ef47546768883016105c52225 (127.8.68.197:45801)
I20240501 20:09:14.329109  8652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915752c40a1a4e6e88118bab3b2f3527" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:14.329358  8652 raft_consensus.cc:3037] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.329651  8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9fa4fb6c4f4ca992e389bf54e1ca" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:14.329658  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915752c40a1a4e6e88118bab3b2f3527" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:14.329813  9176 raft_consensus.cc:3037] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.329831  8650 raft_consensus.cc:3037] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.332038  8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64af9fa4fb6c4f4ca992e389bf54e1ca" candidate_uuid: "4ea5761ef47546768883016105c52225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fb13c94e664ec280fad2968e515d19"
I20240501 20:09:14.332223  8913 raft_consensus.cc:3037] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.332492  8652 raft_consensus.cc:2454] T 915752c40a1a4e6e88118bab3b2f3527 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:14.332556  8650 raft_consensus.cc:2454] T 64af9fa4fb6c4f4ca992e389bf54e1ca P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 1.
I20240501 20:09:14.332903  8848 leader_election.cc:302] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:14.333081  9110 leader_election.cc:302] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea5761ef47546768883016105c52225, a0e6c99bc66b433f9d5cf9340dad0cc0; no voters: 
I20240501 20:09:14.333190 10350 raft_consensus.cc:2781] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:14.333379 10350 raft_consensus.cc:686] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Running, Role: LEADER
I20240501 20:09:14.333418 10057 raft_consensus.cc:2781] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:14.333544 10057 raft_consensus.cc:686] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea5761ef47546768883016105c52225, State: Running, Role: LEADER
I20240501 20:09:14.333688 10350 consensus_queue.cc:237] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } }
I20240501 20:09:14.333791 10057 consensus_queue.cc:237] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } }
I20240501 20:09:14.337473  8505 catalog_manager.cc:5484] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ea5761ef47546768883016105c52225 (127.8.68.197). New cstate: current_term: 1 leader_uuid: "4ea5761ef47546768883016105c52225" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:14.338129  8506 catalog_manager.cc:5484] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fb13c94e664ec280fad2968e515d19 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "02fb13c94e664ec280fad2968e515d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:14.406801 10330 raft_consensus.cc:482] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:14.406966 10330 raft_consensus.cc:504] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:14.407822 10330 leader_election.cc:288] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:14.408223  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c66aafb88274682a73912c1f47e9b03" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:14.408231  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c66aafb88274682a73912c1f47e9b03" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:14.408411  9307 raft_consensus.cc:2454] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:14.408411  9044 raft_consensus.cc:2454] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:14.408819  8716 leader_election.cc:302] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:14.409089 10330 raft_consensus.cc:2781] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:14.409201 10330 raft_consensus.cc:482] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:14.409287 10330 raft_consensus.cc:3037] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.424876  9176 raft_consensus.cc:2454] T 915752c40a1a4e6e88118bab3b2f3527 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:14.424917  8913 raft_consensus.cc:2454] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea5761ef47546768883016105c52225 in term 1.
I20240501 20:09:14.424953 10330 raft_consensus.cc:504] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:14.425740 10330 leader_election.cc:288] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:14.426177  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c66aafb88274682a73912c1f47e9b03" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:14.426235  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c66aafb88274682a73912c1f47e9b03" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:14.426383  9307 raft_consensus.cc:3037] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.426450  9044 raft_consensus.cc:3037] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:14.428323  9307 raft_consensus.cc:2454] T 2c66aafb88274682a73912c1f47e9b03 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:14.428726  8716 leader_election.cc:302] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:14.429046  9044 raft_consensus.cc:2454] T 2c66aafb88274682a73912c1f47e9b03 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:14.429060 10330 raft_consensus.cc:2781] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:14.429230 10330 raft_consensus.cc:686] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:09:14.429577 10330 consensus_queue.cc:237] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:14.432989  8506 catalog_manager.cc:5484] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:14.716619 10350 consensus_queue.cc:1035] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:14.723774 10266 consensus_queue.cc:1035] T 915752c40a1a4e6e88118bab3b2f3527 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:14.808738 10351 consensus_queue.cc:1035] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:14.820353 10357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:14.820664 10357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:09:14.825866 10368 consensus_queue.cc:1035] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:14.831065 10357 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 20:09:14.884941 10330 consensus_queue.cc:1035] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:14.904399 10079 consensus_queue.cc:1035] T 2c66aafb88274682a73912c1f47e9b03 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 253.346 ms
  time per row: 0.126673 ms
I20240501 20:09:15.236099  8505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41588:
name: "rebalance_test_table_11"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:15.247149  9286 tablet_service.cc:1467] Processing CreateTablet for tablet 84f0db662e6e40dab4f50bc10a51b35a (DEFAULT_TABLE table=rebalance_test_table_11 [id=04ecf5f0e4714418aa4b1f9a5c20ed87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.247532  9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f0db662e6e40dab4f50bc10a51b35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.248512  9156 tablet_service.cc:1467] Processing CreateTablet for tablet 4abdd168969845a5a661ec0f4273c2e3 (DEFAULT_TABLE table=rebalance_test_table_11 [id=04ecf5f0e4714418aa4b1f9a5c20ed87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.248749  9024 tablet_service.cc:1467] Processing CreateTablet for tablet 84f0db662e6e40dab4f50bc10a51b35a (DEFAULT_TABLE table=rebalance_test_table_11 [id=04ecf5f0e4714418aa4b1f9a5c20ed87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.248880  9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abdd168969845a5a661ec0f4273c2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.249151  9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f0db662e6e40dab4f50bc10a51b35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.250818  9025 tablet_service.cc:1467] Processing CreateTablet for tablet c94e2d90dcdb47c199c2aa712bd3e300 (DEFAULT_TABLE table=rebalance_test_table_11 [id=04ecf5f0e4714418aa4b1f9a5c20ed87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.251149  9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94e2d90dcdb47c199c2aa712bd3e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.254550  8762 tablet_service.cc:1467] Processing CreateTablet for tablet 84f0db662e6e40dab4f50bc10a51b35a (DEFAULT_TABLE table=rebalance_test_table_11 [id=04ecf5f0e4714418aa4b1f9a5c20ed87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.254756  8763 tablet_service.cc:1467] Processing CreateTablet for tablet c94e2d90dcdb47c199c2aa712bd3e300 (DEFAULT_TABLE table=rebalance_test_table_11 [id=04ecf5f0e4714418aa4b1f9a5c20ed87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.254963  8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f0db662e6e40dab4f50bc10a51b35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.255141  8763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94e2d90dcdb47c199c2aa712bd3e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.258694  8894 tablet_service.cc:1467] Processing CreateTablet for tablet 4abdd168969845a5a661ec0f4273c2e3 (DEFAULT_TABLE table=rebalance_test_table_11 [id=04ecf5f0e4714418aa4b1f9a5c20ed87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.259075  8894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abdd168969845a5a661ec0f4273c2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.263582  9287 tablet_service.cc:1467] Processing CreateTablet for tablet c94e2d90dcdb47c199c2aa712bd3e300 (DEFAULT_TABLE table=rebalance_test_table_11 [id=04ecf5f0e4714418aa4b1f9a5c20ed87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.263936  9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94e2d90dcdb47c199c2aa712bd3e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.267089  8632 tablet_service.cc:1467] Processing CreateTablet for tablet 4abdd168969845a5a661ec0f4273c2e3 (DEFAULT_TABLE table=rebalance_test_table_11 [id=04ecf5f0e4714418aa4b1f9a5c20ed87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.267457  8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abdd168969845a5a661ec0f4273c2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.267772 10427 tablet_bootstrap.cc:492] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:15.272946 10429 tablet_bootstrap.cc:492] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:15.273100 10427 tablet_bootstrap.cc:654] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.275118 10431 tablet_bootstrap.cc:492] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19: Bootstrap starting.
I20240501 20:09:15.275321 10430 tablet_bootstrap.cc:492] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:15.275816 10428 tablet_bootstrap.cc:492] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225: Bootstrap starting.
I20240501 20:09:15.277019 10431 tablet_bootstrap.cc:654] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.277186 10430 tablet_bootstrap.cc:654] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.277772 10429 tablet_bootstrap.cc:654] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.278740 10428 tablet_bootstrap.cc:654] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.283234 10433 tablet_bootstrap.cc:492] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0: Bootstrap starting.
I20240501 20:09:15.285506 10433 tablet_bootstrap.cc:654] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.286722 10430 tablet_bootstrap.cc:492] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:15.286901 10430 ts_tablet_manager.cc:1367] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.007s
I20240501 20:09:15.286943 10427 tablet_bootstrap.cc:492] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:15.287127 10427 ts_tablet_manager.cc:1367] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.019s	user 0.005s	sys 0.000s
I20240501 20:09:15.287688 10430 raft_consensus.cc:348] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.288172 10430 raft_consensus.cc:374] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.288314 10430 raft_consensus.cc:729] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.288404 10431 tablet_bootstrap.cc:492] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19: No bootstrap required, opened a new log
I20240501 20:09:15.288556 10431 ts_tablet_manager.cc:1367] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.003s
I20240501 20:09:15.287868 10427 raft_consensus.cc:348] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.288890 10427 raft_consensus.cc:374] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.288806 10430 consensus_queue.cc:260] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.289049 10427 raft_consensus.cc:729] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.289268 10431 raft_consensus.cc:348] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:15.289283 10427 consensus_queue.cc:260] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.289463 10431 raft_consensus.cc:374] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.289562 10431 raft_consensus.cc:729] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.289759 10431 consensus_queue.cc:260] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:15.290081 10427 ts_tablet_manager.cc:1398] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 20:09:15.290517 10427 tablet_bootstrap.cc:492] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6: Bootstrap starting.
I20240501 20:09:15.290597 10431 ts_tablet_manager.cc:1398] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:09:15.291512 10429 tablet_bootstrap.cc:492] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:15.291695 10429 ts_tablet_manager.cc:1367] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.019s	user 0.006s	sys 0.000s
I20240501 20:09:15.292101 10427 tablet_bootstrap.cc:654] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.292470 10429 raft_consensus.cc:348] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.292718 10429 raft_consensus.cc:374] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.292851 10429 raft_consensus.cc:729] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.293114 10429 consensus_queue.cc:260] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.293618 10428 tablet_bootstrap.cc:492] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225: No bootstrap required, opened a new log
I20240501 20:09:15.293797 10428 ts_tablet_manager.cc:1367] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.000s
I20240501 20:09:15.294615 10429 ts_tablet_manager.cc:1398] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 20:09:15.294570 10428 raft_consensus.cc:348] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:15.294824 10428 raft_consensus.cc:374] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.294931 10429 tablet_bootstrap.cc:492] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Bootstrap starting.
I20240501 20:09:15.295249 10428 raft_consensus.cc:729] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea5761ef47546768883016105c52225, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.295806 10428 consensus_queue.cc:260] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:15.296367 10429 tablet_bootstrap.cc:654] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.296504 10428 ts_tablet_manager.cc:1398] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:15.298494 10433 tablet_bootstrap.cc:492] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0: No bootstrap required, opened a new log
I20240501 20:09:15.298795 10433 ts_tablet_manager.cc:1367] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.001s
I20240501 20:09:15.299547 10433 raft_consensus.cc:348] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:15.299788 10433 raft_consensus.cc:374] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.299938 10433 raft_consensus.cc:729] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e6c99bc66b433f9d5cf9340dad0cc0, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.300220 10433 consensus_queue.cc:260] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:15.300513 10427 tablet_bootstrap.cc:492] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6: No bootstrap required, opened a new log
I20240501 20:09:15.300690 10427 ts_tablet_manager.cc:1367] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.000s
I20240501 20:09:15.301921 10430 ts_tablet_manager.cc:1398] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.015s	user 0.001s	sys 0.001s
I20240501 20:09:15.302749 10427 raft_consensus.cc:348] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.303017 10427 raft_consensus.cc:374] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.303071 10430 tablet_bootstrap.cc:492] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524: Bootstrap starting.
I20240501 20:09:15.303135 10427 raft_consensus.cc:729] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23543295bda14131a70bd31c573454f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.303404 10427 consensus_queue.cc:260] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.304811 10427 ts_tablet_manager.cc:1398] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240501 20:09:15.305554 10433 ts_tablet_manager.cc:1398] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.001s
I20240501 20:09:15.305920 10430 tablet_bootstrap.cc:654] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.310184 10429 tablet_bootstrap.cc:492] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: No bootstrap required, opened a new log
I20240501 20:09:15.310516 10429 ts_tablet_manager.cc:1367] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.004s
I20240501 20:09:15.311057 10429 raft_consensus.cc:348] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.311219 10429 raft_consensus.cc:374] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.311288 10429 raft_consensus.cc:729] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.311475 10430 tablet_bootstrap.cc:492] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524: No bootstrap required, opened a new log
I20240501 20:09:15.311496 10429 consensus_queue.cc:260] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.311604 10430 ts_tablet_manager.cc:1367] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.003s
I20240501 20:09:15.311959 10429 ts_tablet_manager.cc:1398] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:09:15.312161 10079 raft_consensus.cc:482] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:15.312263 10430 raft_consensus.cc:348] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.312311 10079 raft_consensus.cc:504] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.312481 10430 raft_consensus.cc:374] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.312561 10430 raft_consensus.cc:729] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.312819 10430 consensus_queue.cc:260] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.312906 10079 leader_election.cc:288] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:15.313135  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f0db662e6e40dab4f50bc10a51b35a" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:15.313329  9307 raft_consensus.cc:2454] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:15.313386 10430 ts_tablet_manager.cc:1398] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:09:15.313303  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f0db662e6e40dab4f50bc10a51b35a" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524" is_pre_election: true
I20240501 20:09:15.313674  9044 raft_consensus.cc:2454] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 0.
I20240501 20:09:15.313899  8716 leader_election.cc:302] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:15.314386 10079 raft_consensus.cc:2781] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:15.314497 10079 raft_consensus.cc:482] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:15.314582 10079 raft_consensus.cc:3037] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.316807 10079 raft_consensus.cc:504] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.317392 10079 leader_election.cc:288] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123), 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387)
I20240501 20:09:15.317736  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f0db662e6e40dab4f50bc10a51b35a" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:15.317895  9307 raft_consensus.cc:3037] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.317927  9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f0db662e6e40dab4f50bc10a51b35a" candidate_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6c3b236a5d40bf99869152603b5524"
I20240501 20:09:15.318110  9044 raft_consensus.cc:3037] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.319893  9307 raft_consensus.cc:2454] T 84f0db662e6e40dab4f50bc10a51b35a P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:15.320179  8716 leader_election.cc:302] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:15.320288  9044 raft_consensus.cc:2454] T 84f0db662e6e40dab4f50bc10a51b35a P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e66c1f8df41d3a228b1f8b8ece5d8 in term 1.
I20240501 20:09:15.320417 10079 raft_consensus.cc:2781] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:15.320546 10079 raft_consensus.cc:686] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e66c1f8df41d3a228b1f8b8ece5d8, State: Running, Role: LEADER
I20240501 20:09:15.320811 10079 consensus_queue.cc:237] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.323428  8505 catalog_manager.cc:5484] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:15.366816 10235 raft_consensus.cc:482] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:15.366947 10235 raft_consensus.cc:504] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.367512 10235 leader_election.cc:288] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:15.367947  8783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94e2d90dcdb47c199c2aa712bd3e300" candidate_uuid: "3b6c3b236a5d40bf99869152603b5524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" is_pre_election: true
I20240501 20:09:15.368016  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94e2d90dcdb47c199c2aa712bd3e300" candidate_uuid: "3b6c3b236a5d40bf99869152603b5524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6" is_pre_election: true
I20240501 20:09:15.368120  8783 raft_consensus.cc:2454] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6c3b236a5d40bf99869152603b5524 in term 0.
I20240501 20:09:15.368221  9307 raft_consensus.cc:2454] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6c3b236a5d40bf99869152603b5524 in term 0.
I20240501 20:09:15.368508  8978 leader_election.cc:302] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6c3b236a5d40bf99869152603b5524, 3c4e66c1f8df41d3a228b1f8b8ece5d8; no voters: 
I20240501 20:09:15.368738 10235 raft_consensus.cc:2781] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:15.368847 10235 raft_consensus.cc:482] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:15.368927 10235 raft_consensus.cc:3037] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.370854 10235 raft_consensus.cc:504] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.371388 10235 leader_election.cc:288] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [CANDIDATE]: Term 1 election: Requested vote from peers 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923), 23543295bda14131a70bd31c573454f6 (127.8.68.198:39123)
I20240501 20:09:15.371832  8783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94e2d90dcdb47c199c2aa712bd3e300" candidate_uuid: "3b6c3b236a5d40bf99869152603b5524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8"
I20240501 20:09:15.371866  9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94e2d90dcdb47c199c2aa712bd3e300" candidate_uuid: "3b6c3b236a5d40bf99869152603b5524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23543295bda14131a70bd31c573454f6"
I20240501 20:09:15.371991  9307 raft_consensus.cc:3037] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.372051  8783 raft_consensus.cc:3037] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.373687  9307 raft_consensus.cc:2454] T c94e2d90dcdb47c199c2aa712bd3e300 P 23543295bda14131a70bd31c573454f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6c3b236a5d40bf99869152603b5524 in term 1.
I20240501 20:09:15.373991  8783 raft_consensus.cc:2454] T c94e2d90dcdb47c199c2aa712bd3e300 P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6c3b236a5d40bf99869152603b5524 in term 1.
I20240501 20:09:15.374032  8978 leader_election.cc:302] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23543295bda14131a70bd31c573454f6, 3b6c3b236a5d40bf99869152603b5524; no voters: 
I20240501 20:09:15.374305 10235 raft_consensus.cc:2781] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:15.374715 10235 raft_consensus.cc:686] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [term 1 LEADER]: Becoming Leader. State: Replica: 3b6c3b236a5d40bf99869152603b5524, State: Running, Role: LEADER
I20240501 20:09:15.374980 10235 consensus_queue.cc:237] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } }
I20240501 20:09:15.377385  8505 catalog_manager.cc:5484] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196). New cstate: current_term: 1 leader_uuid: "3b6c3b236a5d40bf99869152603b5524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:15.659278 10361 raft_consensus.cc:482] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:15.659466 10361 raft_consensus.cc:504] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:15.660161 10361 leader_election.cc:288] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:15.660578  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abdd168969845a5a661ec0f4273c2e3" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225" is_pre_election: true
I20240501 20:09:15.660802  9176 raft_consensus.cc:2454] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:15.660786  8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abdd168969845a5a661ec0f4273c2e3" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" is_pre_election: true
I20240501 20:09:15.660980  8650 raft_consensus.cc:2454] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 0.
I20240501 20:09:15.661180  8849 leader_election.cc:302] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:15.661526 10361 raft_consensus.cc:2781] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:15.661625 10361 raft_consensus.cc:482] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:15.661706 10361 raft_consensus.cc:3037] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.663477 10361 raft_consensus.cc:504] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:15.664000 10361 leader_election.cc:288] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Requested vote from peers 4ea5761ef47546768883016105c52225 (127.8.68.197:45801), a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491)
I20240501 20:09:15.664299  9176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abdd168969845a5a661ec0f4273c2e3" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea5761ef47546768883016105c52225"
I20240501 20:09:15.664495  9176 raft_consensus.cc:3037] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.664445  8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abdd168969845a5a661ec0f4273c2e3" candidate_uuid: "02fb13c94e664ec280fad2968e515d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0"
I20240501 20:09:15.664631  8650 raft_consensus.cc:3037] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.667097  9176 raft_consensus.cc:2454] T 4abdd168969845a5a661ec0f4273c2e3 P 4ea5761ef47546768883016105c52225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:15.667148  8650 raft_consensus.cc:2454] T 4abdd168969845a5a661ec0f4273c2e3 P a0e6c99bc66b433f9d5cf9340dad0cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fb13c94e664ec280fad2968e515d19 in term 1.
I20240501 20:09:15.667553  8849 leader_election.cc:302] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fb13c94e664ec280fad2968e515d19, 4ea5761ef47546768883016105c52225; no voters: 
I20240501 20:09:15.667822 10266 raft_consensus.cc:2781] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:15.667999 10266 raft_consensus.cc:686] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb13c94e664ec280fad2968e515d19, State: Running, Role: LEADER
I20240501 20:09:15.668272 10266 consensus_queue.cc:237] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } }
I20240501 20:09:15.670935  8505 catalog_manager.cc:5484] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fb13c94e664ec280fad2968e515d19 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "02fb13c94e664ec280fad2968e515d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fb13c94e664ec280fad2968e515d19" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:15.760349 10416 consensus_queue.cc:1035] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6c3b236a5d40bf99869152603b5524" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:15.776405 10451 consensus_queue.cc:1035] T 84f0db662e6e40dab4f50bc10a51b35a P 3c4e66c1f8df41d3a228b1f8b8ece5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:15.885174 10235 consensus_queue.cc:1035] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4e66c1f8df41d3a228b1f8b8ece5d8" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:15.890168 10235 consensus_queue.cc:1035] T c94e2d90dcdb47c199c2aa712bd3e300 P 3b6c3b236a5d40bf99869152603b5524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23543295bda14131a70bd31c573454f6" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:16.038933 10447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:16.039287 10447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:16.047080 10447 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 20:09:16.133623 10361 consensus_queue.cc:1035] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea5761ef47546768883016105c52225" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:16.174540 10361 consensus_queue.cc:1035] T 4abdd168969845a5a661ec0f4273c2e3 P 02fb13c94e664ec280fad2968e515d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e6c99bc66b433f9d5cf9340dad0cc0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:16.246421  8978 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:34923, user_credentials={real_user=slave}} blocked reactor thread for 44876.7us
W20240501 20:09:16.258713  9240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:34923, user_credentials={real_user=slave}} blocked reactor thread for 54381.4us

INSERT report
    rows total: 2000
    time total: 230.336 ms
  time per row: 0.115168 ms
I20240501 20:09:16.905174  8894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:16.910779  9287 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:16.923782  9025 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:16.924683  9156 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:16.940757  8762 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:16.977840  8632 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:17.190191  9287 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:17.191759  9025 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.192186  8762 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:17.194787  9156 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:17.206528  8894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:17.214138  8632 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:17.525463  8894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:17.526185  9287 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:17.554986  9025 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.574200  8762 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:17.577639  9156 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:17.589625  8632 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:17.781733  8894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:17.791468  8762 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:17.807512  9156 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:17.817258  9025 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.822270  9287 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:17.863490  8632 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:18.072543  9156 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:18.073213  9025 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.081588  8632 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:18.092731  8762 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:18.105541  9287 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:18.114732  8894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:18.481139  8894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:18.481786  9025 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.485059  8762 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:09:18.497263  9287 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:18.497431  9156 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:18.523698  8632 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 7f64f846da674e90b273568281328ac0 | 127.8.68.254:39131 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                    Value                                                                                     |         Tags          |         Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                          | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                          | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                         | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                            | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                         | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                         | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                         | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                           | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                          | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.8.68.212:46533 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 02fb13c94e664ec280fad2968e515d19 | 127.8.68.195:44647 | HEALTHY | /B       |       9        |       0
 23543295bda14131a70bd31c573454f6 | 127.8.68.198:39123 | HEALTHY | /C       |       6        |       0
 3b6c3b236a5d40bf99869152603b5524 | 127.8.68.196:38387 | HEALTHY | /B       |       2        |       0
 3c4e66c1f8df41d3a228b1f8b8ece5d8 | 127.8.68.194:34923 | HEALTHY | /A       |      11        |       0
 4ea5761ef47546768883016105c52225 | 127.8.68.197:45801 | HEALTHY | /C       |       4        |       0
 a0e6c99bc66b433f9d5cf9340dad0cc0 | 127.8.68.193:45491 | HEALTHY | /A       |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 /A       |       2
 /B       |       2
 /C       |       2

Unusual flags for Tablet Server:
                   Flag                   |                                                                                  Value                                                                                   |         Tags          |      Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                      | experimental          | all 6 server(s) checked
 local_ip_for_outbound_sockets            | 127.8.68.193                                                                                                                                                             | experimental          | 127.8.68.193:45491
 local_ip_for_outbound_sockets            | 127.8.68.194                                                                                                                                                             | experimental          | 127.8.68.194:34923
 local_ip_for_outbound_sockets            | 127.8.68.195                                                                                                                                                             | experimental          | 127.8.68.195:44647
 local_ip_for_outbound_sockets            | 127.8.68.196                                                                                                                                                             | experimental          | 127.8.68.196:38387
 local_ip_for_outbound_sockets            | 127.8.68.197                                                                                                                                                             | experimental          | 127.8.68.197:45801
 local_ip_for_outbound_sockets            | 127.8.68.198                                                                                                                                                             | experimental          | 127.8.68.198:39123
 never_fsync                              | true                                                                                                                                                                     | unsafe,advanced       | all 6 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                        | unsafe,hidden         | all 6 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                     | experimental,advanced | all 6 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                     | unsafe                | all 6 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                       | hidden                | all 6 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.8.68.193:45491
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.8.68.194:34923
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.8.68.195:44647
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.8.68.196:38387
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.8.68.197:45801
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.8.68.198:39123

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.8.68.212:46533 | all 6 server(s) checked
 time_source         | builtin            | all 6 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 7 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name           | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
 rebalance_test_table_0  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_10 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_11 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_5  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_6  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_7  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_8  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_9  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 18
 First Quartile | 18
 Median         | 18
 Third Quartile | 18
 Maximum        | 18

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 6
 Tables         | 12
 Tablets        | 36
 Replicas       | 108

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240501 20:09:18.584231  8467 log_verifier.cc:126] Checking tablet 009d7f2fece0434795136dda912beb51
I20240501 20:09:18.621863  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 009d7f2fece0434795136dda912beb51
I20240501 20:09:18.622016  8467 log_verifier.cc:126] Checking tablet 122a2e00098f4ca9b91509a85c6c5860
I20240501 20:09:18.658293  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 122a2e00098f4ca9b91509a85c6c5860
I20240501 20:09:18.658411  8467 log_verifier.cc:126] Checking tablet 14969b6bae9c44a19f774abc330216ec
I20240501 20:09:18.692189  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 14969b6bae9c44a19f774abc330216ec
I20240501 20:09:18.692293  8467 log_verifier.cc:126] Checking tablet 26ad523451b64159877795981558e17c
I20240501 20:09:18.728701  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 26ad523451b64159877795981558e17c
I20240501 20:09:18.728803  8467 log_verifier.cc:126] Checking tablet 2c66aafb88274682a73912c1f47e9b03
I20240501 20:09:18.766202  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2c66aafb88274682a73912c1f47e9b03
I20240501 20:09:18.766336  8467 log_verifier.cc:126] Checking tablet 3cc72717208547689b45b9c789c8f3f9
I20240501 20:09:18.804848  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3cc72717208547689b45b9c789c8f3f9
I20240501 20:09:18.804965  8467 log_verifier.cc:126] Checking tablet 3ee63f84c8ff4bdd86c2302163f25e6c
I20240501 20:09:18.843068  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3ee63f84c8ff4bdd86c2302163f25e6c
I20240501 20:09:18.843166  8467 log_verifier.cc:126] Checking tablet 3f0f18068e024139a123f055c29b36b5
I20240501 20:09:18.882644  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3f0f18068e024139a123f055c29b36b5
I20240501 20:09:18.882750  8467 log_verifier.cc:126] Checking tablet 42a721314bd4471eb2ffe27dc8acdf5c
I20240501 20:09:18.922432  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 42a721314bd4471eb2ffe27dc8acdf5c
I20240501 20:09:18.922540  8467 log_verifier.cc:126] Checking tablet 43bf7725b0b94c2eac42fca017b10a2f
I20240501 20:09:18.960852  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 43bf7725b0b94c2eac42fca017b10a2f
I20240501 20:09:18.960986  8467 log_verifier.cc:126] Checking tablet 4abdd168969845a5a661ec0f4273c2e3
I20240501 20:09:19.001253  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4abdd168969845a5a661ec0f4273c2e3
I20240501 20:09:19.001350  8467 log_verifier.cc:126] Checking tablet 514b5a047b694b878818b43f2e8c2f80
I20240501 20:09:19.040067  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 514b5a047b694b878818b43f2e8c2f80
I20240501 20:09:19.040165  8467 log_verifier.cc:126] Checking tablet 64af9fa4fb6c4f4ca992e389bf54e1ca
I20240501 20:09:19.085103  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 64af9fa4fb6c4f4ca992e389bf54e1ca
I20240501 20:09:19.085199  8467 log_verifier.cc:126] Checking tablet 66a87a4b36cc40998d8e89ad0fb3265b
I20240501 20:09:19.123548  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 66a87a4b36cc40998d8e89ad0fb3265b
I20240501 20:09:19.123646  8467 log_verifier.cc:126] Checking tablet 6b4e4fa3e651453ba0344d5dc2607700
I20240501 20:09:19.166507  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6b4e4fa3e651453ba0344d5dc2607700
I20240501 20:09:19.166641  8467 log_verifier.cc:126] Checking tablet 7f1bca656e234c68b8cfd0b1f2523f86
I20240501 20:09:19.211337  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7f1bca656e234c68b8cfd0b1f2523f86
I20240501 20:09:19.211436  8467 log_verifier.cc:126] Checking tablet 83c877a1b0d04d638854d654c46da343
I20240501 20:09:19.259577  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 83c877a1b0d04d638854d654c46da343
I20240501 20:09:19.259712  8467 log_verifier.cc:126] Checking tablet 84f0db662e6e40dab4f50bc10a51b35a
I20240501 20:09:19.300972  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 84f0db662e6e40dab4f50bc10a51b35a
I20240501 20:09:19.301111  8467 log_verifier.cc:126] Checking tablet 863b82c7886746d2a3932c9cc6ce9019
I20240501 20:09:19.341601  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 863b82c7886746d2a3932c9cc6ce9019
I20240501 20:09:19.341706  8467 log_verifier.cc:126] Checking tablet 915752c40a1a4e6e88118bab3b2f3527
I20240501 20:09:19.381937  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 915752c40a1a4e6e88118bab3b2f3527
I20240501 20:09:19.382035  8467 log_verifier.cc:126] Checking tablet 9e288e6d1c404303a044cbae80e13491
I20240501 20:09:19.420969  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9e288e6d1c404303a044cbae80e13491
I20240501 20:09:19.421073  8467 log_verifier.cc:126] Checking tablet a013dd2c5905470282f770673d0c1cef
I20240501 20:09:19.459510  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a013dd2c5905470282f770673d0c1cef
I20240501 20:09:19.459645  8467 log_verifier.cc:126] Checking tablet b82d800828b3481e9bda31ee121a6588
I20240501 20:09:19.498777  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b82d800828b3481e9bda31ee121a6588
I20240501 20:09:19.498905  8467 log_verifier.cc:126] Checking tablet b88d93c63032451eb0e6904aaac0001b
I20240501 20:09:19.539737  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b88d93c63032451eb0e6904aaac0001b
I20240501 20:09:19.539876  8467 log_verifier.cc:126] Checking tablet c8b7690f910048ab9d7ca0e9c8c57006
I20240501 20:09:19.577044  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c8b7690f910048ab9d7ca0e9c8c57006
I20240501 20:09:19.577155  8467 log_verifier.cc:126] Checking tablet c94e2d90dcdb47c199c2aa712bd3e300
I20240501 20:09:19.616231  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c94e2d90dcdb47c199c2aa712bd3e300
I20240501 20:09:19.616334  8467 log_verifier.cc:126] Checking tablet d205586c09e8445f91ea63579e8b8ded
I20240501 20:09:19.655313  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d205586c09e8445f91ea63579e8b8ded
I20240501 20:09:19.655427  8467 log_verifier.cc:126] Checking tablet da583015bf3e4d3b99c32d653dc8131b
I20240501 20:09:19.694936  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet da583015bf3e4d3b99c32d653dc8131b
I20240501 20:09:19.695045  8467 log_verifier.cc:126] Checking tablet dd0b0c1f3bf0404e99a6ea9c73bef90e
I20240501 20:09:19.734091  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dd0b0c1f3bf0404e99a6ea9c73bef90e
I20240501 20:09:19.734234  8467 log_verifier.cc:126] Checking tablet dd5acbf391ca4f7281677b2f2584a7f9
I20240501 20:09:19.774190  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dd5acbf391ca4f7281677b2f2584a7f9
I20240501 20:09:19.774331  8467 log_verifier.cc:126] Checking tablet e3fa355226244b9ba3c1482c479ab94b
I20240501 20:09:19.812521  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e3fa355226244b9ba3c1482c479ab94b
I20240501 20:09:19.812618  8467 log_verifier.cc:126] Checking tablet e93c8da097b64b6ead9bbb1168fd35e7
I20240501 20:09:19.852120  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e93c8da097b64b6ead9bbb1168fd35e7
I20240501 20:09:19.852236  8467 log_verifier.cc:126] Checking tablet ed26eb2563fd44748ff4b080e68c922d
I20240501 20:09:19.890376  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ed26eb2563fd44748ff4b080e68c922d
I20240501 20:09:19.890486  8467 log_verifier.cc:126] Checking tablet ed8378c3ec35479ea1821e027c37866f
I20240501 20:09:19.929136  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ed8378c3ec35479ea1821e027c37866f
I20240501 20:09:19.929239  8467 log_verifier.cc:126] Checking tablet f706a5f9c68a4a9e8cf7988e2ee71c63
I20240501 20:09:19.968713  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f706a5f9c68a4a9e8cf7988e2ee71c63
I20240501 20:09:19.968883  8467 log_verifier.cc:126] Checking tablet fee23b1bdb8249b68a8464928344277f
I20240501 20:09:20.007620  8467 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fee23b1bdb8249b68a8464928344277f
I20240501 20:09:20.029923  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 8570
I20240501 20:09:20.038022  8692 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:20.182957  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 8570
W20240501 20:09:20.203692  8847 connection.cc:535] server connection from 127.8.68.193:39227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:09:20.204226  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 8700
I20240501 20:09:20.209945  8822 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:09:20.221150  9110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111)
W20240501 20:09:20.222707  9110 consensus_peers.cc:485] T da583015bf3e4d3b99c32d653dc8131b P 4ea5761ef47546768883016105c52225 -> Peer a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491): Couldn't send request to peer a0e6c99bc66b433f9d5cf9340dad0cc0. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.253760  9241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111)
W20240501 20:09:20.255115  9241 consensus_peers.cc:485] T b88d93c63032451eb0e6904aaac0001b P 23543295bda14131a70bd31c573454f6 -> Peer a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491): Couldn't send request to peer a0e6c99bc66b433f9d5cf9340dad0cc0. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.265378  9110 consensus_peers.cc:485] T 64af9fa4fb6c4f4ca992e389bf54e1ca P 4ea5761ef47546768883016105c52225 -> Peer a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491): Couldn't send request to peer a0e6c99bc66b433f9d5cf9340dad0cc0. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.335021  8848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111)
W20240501 20:09:20.336483  8848 consensus_peers.cc:485] T e93c8da097b64b6ead9bbb1168fd35e7 P 02fb13c94e664ec280fad2968e515d19 -> Peer a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491): Couldn't send request to peer a0e6c99bc66b433f9d5cf9340dad0cc0. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.339836  8848 consensus_peers.cc:485] T dd5acbf391ca4f7281677b2f2584a7f9 P 02fb13c94e664ec280fad2968e515d19 -> Peer a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491): Couldn't send request to peer a0e6c99bc66b433f9d5cf9340dad0cc0. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:20.345610  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 8700
W20240501 20:09:20.348824  9110 consensus_peers.cc:485] T 514b5a047b694b878818b43f2e8c2f80 P 4ea5761ef47546768883016105c52225 -> Peer a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491): Couldn't send request to peer a0e6c99bc66b433f9d5cf9340dad0cc0. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.365473  8497 connection.cc:535] server connection from 127.8.68.194:56415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:09:20.365520  9242 connection.cc:535] server connection from 127.8.68.194:35863 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:09:20.365803  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 8831
I20240501 20:09:20.371609  8954 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:09:20.371624  9240 consensus_peers.cc:485] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 -> Peer 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923): Couldn't send request to peer 3c4e66c1f8df41d3a228b1f8b8ece5d8. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:34923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.392112  9110 consensus_peers.cc:485] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 -> Peer a0e6c99bc66b433f9d5cf9340dad0cc0 (127.8.68.193:45491): Couldn't send request to peer a0e6c99bc66b433f9d5cf9340dad0cc0. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.433333  8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.194:34923: connect: Connection refused (error 111)
W20240501 20:09:20.434839  8978 consensus_peers.cc:485] T 9e288e6d1c404303a044cbae80e13491 P 3b6c3b236a5d40bf99869152603b5524 -> Peer 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923): Couldn't send request to peer 3c4e66c1f8df41d3a228b1f8b8ece5d8. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:34923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.473980  9240 consensus_peers.cc:485] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 -> Peer 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923): Couldn't send request to peer 3c4e66c1f8df41d3a228b1f8b8ece5d8. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:34923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.489166  9240 consensus_peers.cc:485] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 -> Peer 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923): Couldn't send request to peer 3c4e66c1f8df41d3a228b1f8b8ece5d8. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:34923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:20.502177  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 8831
W20240501 20:09:20.522508  9111 connection.cc:535] server connection from 127.8.68.195:35769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:09:20.522922  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 8962
W20240501 20:09:20.525434  9109 consensus_peers.cc:485] T f706a5f9c68a4a9e8cf7988e2ee71c63 P 4ea5761ef47546768883016105c52225 -> Peer 02fb13c94e664ec280fad2968e515d19 (127.8.68.195:44647): Couldn't send request to peer 02fb13c94e664ec280fad2968e515d19. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.195:44647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:20.529834  9085 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:09:20.577145  9240 consensus_peers.cc:485] T 863b82c7886746d2a3932c9cc6ce9019 P 23543295bda14131a70bd31c573454f6 -> Peer 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923): Couldn't send request to peer 3c4e66c1f8df41d3a228b1f8b8ece5d8. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:34923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:20.660565  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 8962
I20240501 20:09:20.680266  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 9093
I20240501 20:09:20.685912  9215 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:09:20.688732  9240 consensus_peers.cc:485] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 -> Peer 3c4e66c1f8df41d3a228b1f8b8ece5d8 (127.8.68.194:34923): Couldn't send request to peer 3c4e66c1f8df41d3a228b1f8b8ece5d8. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:34923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.700306  9241 consensus_peers.cc:485] T d205586c09e8445f91ea63579e8b8ded P 23543295bda14131a70bd31c573454f6 -> Peer 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387): Couldn't send request to peer 3b6c3b236a5d40bf99869152603b5524. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.196:38387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.736876  9241 consensus_peers.cc:485] T 42a721314bd4471eb2ffe27dc8acdf5c P 23543295bda14131a70bd31c573454f6 -> Peer 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387): Couldn't send request to peer 3b6c3b236a5d40bf99869152603b5524. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.196:38387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.740731  9241 consensus_peers.cc:485] T 009d7f2fece0434795136dda912beb51 P 23543295bda14131a70bd31c573454f6 -> Peer 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387): Couldn't send request to peer 3b6c3b236a5d40bf99869152603b5524. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.196:38387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:20.798516  9241 consensus_peers.cc:485] T ed26eb2563fd44748ff4b080e68c922d P 23543295bda14131a70bd31c573454f6 -> Peer 3b6c3b236a5d40bf99869152603b5524 (127.8.68.196:38387): Couldn't send request to peer 3b6c3b236a5d40bf99869152603b5524. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.196:38387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:20.813023  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 9093
I20240501 20:09:20.832789  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 9224
I20240501 20:09:20.838320  9347 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:20.974506  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 9224
W20240501 20:09:20.994755  8497 connection.cc:535] server connection from 127.8.68.198:59105 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:09:20.995096  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 8479
I20240501 20:09:20.995990  8540 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:21.141919  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 8479
2024-05-01T20:09:21Z chronyd exiting
[       OK ] LocationAwareRebalancingBasicTest.Basic (30809 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (30809 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240501 20:09:21.272099  8467 test_util.cc:268] Using random seed: 267035900
I20240501 20:09:21.273802  8467 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:09:21.273901  8467 ts_itest-base.cc:113] --------------
I20240501 20:09:21.273960  8467 ts_itest-base.cc:114] 5 tablet servers
I20240501 20:09:21.274030  8467 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:09:21.274078  8467 ts_itest-base.cc:116] --------------
2024-05-01T20:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:09:21Z Disabled control of system clock
I20240501 20:09:21.305977  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:41155
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:41155
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:09:21.634594 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:21.634856 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:21.634954 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:21.641984 10836 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:21.642087 10836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:09:21.642127 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:21.642182 10836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:09:21.642226 10836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:09:21.650650 10836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:41155
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:21.651762 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:21.653321 10836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:21.664137 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:21.664489 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:21.664829 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:21.665421 10836 server_base.cc:1034] running on GCE node
I20240501 20:09:21.666059 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:21.667333 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:21.668555 10836 hybrid_clock.cc:648] HybridClock initialized: now 1714594161668508 us; error 37 us; skew 500 ppm
I20240501 20:09:21.668887 10836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:21.670930 10836 webserver.cc:460] Webserver started at http://127.8.68.254:45503/ using document root <none> and password file <none>
I20240501 20:09:21.671553 10836 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:21.671656 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:21.671957 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:21.674472 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "66b04ec6601b42c5942a7a1580bf5449"
format_stamp: "Formatted at 2024-05-01 20:09:21 on dist-test-slave-08p9"
I20240501 20:09:21.675212 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "66b04ec6601b42c5942a7a1580bf5449"
format_stamp: "Formatted at 2024-05-01 20:09:21 on dist-test-slave-08p9"
I20240501 20:09:21.679776 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:09:21.682768 10850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:21.683897 10836 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:09:21.684048 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "66b04ec6601b42c5942a7a1580bf5449"
format_stamp: "Formatted at 2024-05-01 20:09:21 on dist-test-slave-08p9"
I20240501 20:09:21.684216 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:21.703819 10836 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:21.704463 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:21.704690 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:21.720319 10836 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:41155
I20240501 20:09:21.720329 10901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:41155 every 8 connection(s)
I20240501 20:09:21.721397 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:09:21.724573 10902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:21.726002  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 10836
I20240501 20:09:21.726228  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:09:21.734797 10902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449: Bootstrap starting.
I20240501 20:09:21.737423 10902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:21.738404 10902 log.cc:824] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:21.740588 10902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449: No bootstrap required, opened a new log
I20240501 20:09:21.746003 10902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b04ec6601b42c5942a7a1580bf5449" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 41155 } }
I20240501 20:09:21.746291 10902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:21.746387 10902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b04ec6601b42c5942a7a1580bf5449, State: Initialized, Role: FOLLOWER
I20240501 20:09:21.746929 10902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b04ec6601b42c5942a7a1580bf5449" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 41155 } }
I20240501 20:09:21.747112 10902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:21.747215 10902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:21.747344 10902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:21.749131 10902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b04ec6601b42c5942a7a1580bf5449" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 41155 } }
I20240501 20:09:21.749533 10902 leader_election.cc:302] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b04ec6601b42c5942a7a1580bf5449; no voters: 
I20240501 20:09:21.750038 10902 leader_election.cc:288] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:09:21.750142 10907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:21.750749 10907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [term 1 LEADER]: Becoming Leader. State: Replica: 66b04ec6601b42c5942a7a1580bf5449, State: Running, Role: LEADER
I20240501 20:09:21.751235 10907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b04ec6601b42c5942a7a1580bf5449" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 41155 } }
I20240501 20:09:21.751679 10902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:09:21.753795 10908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b04ec6601b42c5942a7a1580bf5449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b04ec6601b42c5942a7a1580bf5449" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 41155 } } }
I20240501 20:09:21.754356 10908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:21.753789 10909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b04ec6601b42c5942a7a1580bf5449. Latest consensus state: current_term: 1 leader_uuid: "66b04ec6601b42c5942a7a1580bf5449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b04ec6601b42c5942a7a1580bf5449" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 41155 } } }
I20240501 20:09:21.754787 10909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:21.755789 10915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:09:21.760553 10915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:09:21.767606 10915 catalog_manager.cc:1347] Generated new cluster ID: fce20ef1209d4563991fae5c083619d4
I20240501 20:09:21.767701 10915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:09:21.788671 10915 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:09:21.789783 10915 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:09:21.797783 10915 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66b04ec6601b42c5942a7a1580bf5449: Generated new TSK 0
I20240501 20:09:21.798604 10915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:09:21.812906  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--builtin_ntp_servers=127.8.68.212:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:09:22.118103 10926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:22.118372 10926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:22.118508 10926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:22.125466 10926 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:22.125571 10926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:22.125717 10926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240501 20:09:22.134032 10926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:22.135226 10926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:22.136752 10926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:22.148119 10931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.148664 10932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.149147 10934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:22.149698 10926 server_base.cc:1034] running on GCE node
I20240501 20:09:22.150274 10926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:22.151121 10926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:22.152325 10926 hybrid_clock.cc:648] HybridClock initialized: now 1714594162152273 us; error 42 us; skew 500 ppm
I20240501 20:09:22.152611 10926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:22.154632 10926 webserver.cc:460] Webserver started at http://127.8.68.193:38671/ using document root <none> and password file <none>
I20240501 20:09:22.155268 10926 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:22.155369 10926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:22.155658 10926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:22.157895 10926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-08p9"
I20240501 20:09:22.158615 10926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-08p9"
I20240501 20:09:22.163077 10926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240501 20:09:22.165979 10940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:22.167141 10926 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:09:22.167273 10926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-08p9"
I20240501 20:09:22.167443 10926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:22.183207 10926 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:22.183852 10926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:22.184089 10926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:22.184875 10926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:22.186172 10926 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:22.186262 10926 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:22.186381 10926 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:22.186452 10926 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:22.215927 10926 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:38105
I20240501 20:09:22.215982 11052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:38105 every 8 connection(s)
I20240501 20:09:22.217047 10926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:09:22.221601  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 10926
I20240501 20:09:22.221791  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:09:22.224910  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--builtin_ntp_servers=127.8.68.212:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:22.228937 11053 heartbeater.cc:347] Connected to a master server at 127.8.68.254:41155
I20240501 20:09:22.229177 11053 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:22.229880 11053 heartbeater.cc:511] Master 127.8.68.254:41155 requested a full tablet report, sending...
I20240501 20:09:22.231757 10867 ts_manager.cc:194] Registered new tserver with Master: 2eb64f4d9d8f4232a3beec44e45ccb3a (127.8.68.193:38105)
I20240501 20:09:22.233178 10867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:50669
W20240501 20:09:22.530078 11056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:22.530334 11056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:22.530449 11056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:22.538637 11056 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:22.538748 11056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:22.538866 11056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240501 20:09:22.547574 11056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:22.548568 11056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:22.550042 11056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:22.561307 11064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.561694 11061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.562144 11062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:22.562623 11056 server_base.cc:1034] running on GCE node
I20240501 20:09:22.563158 11056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:22.563962 11056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:22.565152 11056 hybrid_clock.cc:648] HybridClock initialized: now 1714594162565110 us; error 38 us; skew 500 ppm
I20240501 20:09:22.565433 11056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:22.567431 11056 webserver.cc:460] Webserver started at http://127.8.68.194:43849/ using document root <none> and password file <none>
I20240501 20:09:22.568063 11056 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:22.568169 11056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:22.568459 11056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:22.570793 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "05a79835c84349a0b37cd7ce1c5fa1fa"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-08p9"
I20240501 20:09:22.571517 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "05a79835c84349a0b37cd7ce1c5fa1fa"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-08p9"
I20240501 20:09:22.576148 11056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 20:09:22.579054 11070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:22.580155 11056 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:09:22.580297 11056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "05a79835c84349a0b37cd7ce1c5fa1fa"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-08p9"
I20240501 20:09:22.580471 11056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:22.602155 11056 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:22.602861 11056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:22.603104 11056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:22.603899 11056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:22.605208 11056 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:22.605304 11056 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:22.605405 11056 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:22.605473 11056 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:22.632987 11056 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:40909
I20240501 20:09:22.633057 11182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:40909 every 8 connection(s)
I20240501 20:09:22.634095 11056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:09:22.635617  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 11056
I20240501 20:09:22.635804  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:09:22.638970  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--builtin_ntp_servers=127.8.68.212:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:22.646548 11183 heartbeater.cc:347] Connected to a master server at 127.8.68.254:41155
I20240501 20:09:22.646818 11183 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:22.647644 11183 heartbeater.cc:511] Master 127.8.68.254:41155 requested a full tablet report, sending...
I20240501 20:09:22.649012 10867 ts_manager.cc:194] Registered new tserver with Master: 05a79835c84349a0b37cd7ce1c5fa1fa (127.8.68.194:40909)
I20240501 20:09:22.649799 10867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:44995
W20240501 20:09:22.936913 11186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:22.937170 11186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:22.937260 11186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:22.944257 11186 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:22.944353 11186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:22.944466 11186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240501 20:09:22.952836 11186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:22.953931 11186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:22.955514 11186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:22.966470 11191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.966872 11192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.967341 11194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:22.967849 11186 server_base.cc:1034] running on GCE node
I20240501 20:09:22.968348 11186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:22.969175 11186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:22.970355 11186 hybrid_clock.cc:648] HybridClock initialized: now 1714594162970318 us; error 30 us; skew 500 ppm
I20240501 20:09:22.970625 11186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:22.972643 11186 webserver.cc:460] Webserver started at http://127.8.68.195:39943/ using document root <none> and password file <none>
I20240501 20:09:22.973286 11186 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:22.973393 11186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:22.973681 11186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:22.976002 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "58dc38b7a43240e68f3d9b7fe6c40191"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-08p9"
I20240501 20:09:22.976666 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "58dc38b7a43240e68f3d9b7fe6c40191"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-08p9"
I20240501 20:09:22.981117 11186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240501 20:09:22.984133 11200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:22.985381 11186 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240501 20:09:22.985528 11186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "58dc38b7a43240e68f3d9b7fe6c40191"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-08p9"
I20240501 20:09:22.985697 11186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:23.023736 11186 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:23.024523 11186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:23.024772 11186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:23.025665 11186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:23.027086 11186 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:23.027170 11186 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:23.027271 11186 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:23.027340 11186 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:23.055260 11186 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:41409
I20240501 20:09:23.055337 11312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:41409 every 8 connection(s)
I20240501 20:09:23.056445 11186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:09:23.059974  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 11186
I20240501 20:09:23.060139  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:09:23.063290  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--builtin_ntp_servers=127.8.68.212:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:23.068341 11313 heartbeater.cc:347] Connected to a master server at 127.8.68.254:41155
I20240501 20:09:23.068594 11313 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:23.069298 11313 heartbeater.cc:511] Master 127.8.68.254:41155 requested a full tablet report, sending...
I20240501 20:09:23.070603 10867 ts_manager.cc:194] Registered new tserver with Master: 58dc38b7a43240e68f3d9b7fe6c40191 (127.8.68.195:41409)
I20240501 20:09:23.071368 10867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:58069
I20240501 20:09:23.235630 11053 heartbeater.cc:503] Master 127.8.68.254:41155 was elected leader, sending a full tablet report...
W20240501 20:09:23.374528 11316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:23.374763 11316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:23.374883 11316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:23.382076 11316 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:23.382177 11316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:23.382364 11316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:09:23.390882 11316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:23.391960 11316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:23.393467 11316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:23.404469 11324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.404958 11322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.405380 11321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:23.405956 11316 server_base.cc:1034] running on GCE node
I20240501 20:09:23.406499 11316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:23.407307 11316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:23.408495 11316 hybrid_clock.cc:648] HybridClock initialized: now 1714594163408459 us; error 33 us; skew 500 ppm
I20240501 20:09:23.408778 11316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:23.410861 11316 webserver.cc:460] Webserver started at http://127.8.68.196:41143/ using document root <none> and password file <none>
I20240501 20:09:23.411516 11316 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:23.411623 11316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:23.411953 11316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:23.414260 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6058e29fedba4c69a8725ad7f505e378"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-08p9"
I20240501 20:09:23.414988 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6058e29fedba4c69a8725ad7f505e378"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-08p9"
I20240501 20:09:23.419610 11316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:09:23.422729 11330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:23.423869 11316 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:09:23.424016 11316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6058e29fedba4c69a8725ad7f505e378"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-08p9"
I20240501 20:09:23.424185 11316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:23.446453 11316 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:23.447101 11316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:23.447332 11316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:23.448137 11316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:23.449424 11316 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:23.449512 11316 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:23.449610 11316 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:23.449677 11316 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:23.478089 11316 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:38399
I20240501 20:09:23.478143 11442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:38399 every 8 connection(s)
I20240501 20:09:23.479200 11316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:09:23.483089  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 11316
I20240501 20:09:23.483300  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:09:23.486809  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--builtin_ntp_servers=127.8.68.212:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:23.491765 11443 heartbeater.cc:347] Connected to a master server at 127.8.68.254:41155
I20240501 20:09:23.492080 11443 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:23.492992 11443 heartbeater.cc:511] Master 127.8.68.254:41155 requested a full tablet report, sending...
I20240501 20:09:23.494414 10867 ts_manager.cc:194] Registered new tserver with Master: 6058e29fedba4c69a8725ad7f505e378 (127.8.68.196:38399)
I20240501 20:09:23.495158 10867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:33113
I20240501 20:09:23.652503 11183 heartbeater.cc:503] Master 127.8.68.254:41155 was elected leader, sending a full tablet report...
W20240501 20:09:23.791364 11446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:23.791597 11446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:23.791695 11446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:23.798632 11446 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:23.798727 11446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:23.798836 11446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:09:23.807090 11446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:23.808106 11446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:23.809594 11446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:23.820581 11451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.820946 11452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.821523 11454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:23.822034 11446 server_base.cc:1034] running on GCE node
I20240501 20:09:23.822607 11446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:23.823441 11446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:23.824631 11446 hybrid_clock.cc:648] HybridClock initialized: now 1714594163824590 us; error 37 us; skew 500 ppm
I20240501 20:09:23.824898 11446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:23.826938 11446 webserver.cc:460] Webserver started at http://127.8.68.197:39807/ using document root <none> and password file <none>
I20240501 20:09:23.827597 11446 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:23.827693 11446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:23.828001 11446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:23.830317 11446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "87ff7e2623c84f29aa73f663e72ca1f1"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-08p9"
I20240501 20:09:23.831012 11446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "87ff7e2623c84f29aa73f663e72ca1f1"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-08p9"
I20240501 20:09:23.835417 11446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240501 20:09:23.838349 11460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:23.839468 11446 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:09:23.839599 11446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "87ff7e2623c84f29aa73f663e72ca1f1"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-08p9"
I20240501 20:09:23.839757 11446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:23.858112 11446 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:23.858798 11446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:23.859022 11446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:23.859793 11446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:23.861032 11446 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:23.861115 11446 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:23.861208 11446 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:23.861272 11446 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:23.887459 11446 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:40333
I20240501 20:09:23.887527 11572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:40333 every 8 connection(s)
I20240501 20:09:23.888598 11446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:09:23.897104  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 11446
I20240501 20:09:23.897261  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:09:23.899545 11573 heartbeater.cc:347] Connected to a master server at 127.8.68.254:41155
I20240501 20:09:23.899792 11573 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:23.900456 11573 heartbeater.cc:511] Master 127.8.68.254:41155 requested a full tablet report, sending...
I20240501 20:09:23.901638 10867 ts_manager.cc:194] Registered new tserver with Master: 87ff7e2623c84f29aa73f663e72ca1f1 (127.8.68.197:40333)
I20240501 20:09:23.902402 10867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:56713
I20240501 20:09:23.912488  8467 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:09:23.926699 10867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36070:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240501 20:09:23.955204 11248 tablet_service.cc:1467] Processing CreateTablet for tablet 241afb3dadd747e991a08b17f5d7a581 (DEFAULT_TABLE table=TestTable [id=26e480b2a0794a17b1f5ed7c088dae7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.955204 11378 tablet_service.cc:1467] Processing CreateTablet for tablet 241afb3dadd747e991a08b17f5d7a581 (DEFAULT_TABLE table=TestTable [id=26e480b2a0794a17b1f5ed7c088dae7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.956673 11378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241afb3dadd747e991a08b17f5d7a581. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.958946 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241afb3dadd747e991a08b17f5d7a581. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.958449 10988 tablet_service.cc:1467] Processing CreateTablet for tablet 241afb3dadd747e991a08b17f5d7a581 (DEFAULT_TABLE table=TestTable [id=26e480b2a0794a17b1f5ed7c088dae7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:23.959689 10988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241afb3dadd747e991a08b17f5d7a581. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:23.968468 11591 tablet_bootstrap.cc:492] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191: Bootstrap starting.
I20240501 20:09:23.971242 11591 tablet_bootstrap.cc:654] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.972852 11593 tablet_bootstrap.cc:492] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a: Bootstrap starting.
I20240501 20:09:23.973223 11591 log.cc:824] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:23.974470 11592 tablet_bootstrap.cc:492] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378: Bootstrap starting.
I20240501 20:09:23.976485 11593 tablet_bootstrap.cc:654] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.977195 11592 tablet_bootstrap.cc:654] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:23.977701 11593 log.cc:824] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:23.978152 11592 log.cc:824] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:23.980422 11593 tablet_bootstrap.cc:492] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a: No bootstrap required, opened a new log
I20240501 20:09:23.980695 11593 ts_tablet_manager.cc:1367] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240501 20:09:23.980726 11591 tablet_bootstrap.cc:492] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191: No bootstrap required, opened a new log
I20240501 20:09:23.981212 11591 ts_tablet_manager.cc:1367] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191: Time spent bootstrapping tablet: real 0.013s	user 0.007s	sys 0.004s
I20240501 20:09:23.982590 11592 tablet_bootstrap.cc:492] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378: No bootstrap required, opened a new log
I20240501 20:09:23.982844 11592 ts_tablet_manager.cc:1367] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240501 20:09:23.988821 11592 raft_consensus.cc:348] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } }
I20240501 20:09:23.989269 11592 raft_consensus.cc:374] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.989400 11592 raft_consensus.cc:729] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6058e29fedba4c69a8725ad7f505e378, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.989154 11593 raft_consensus.cc:348] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } }
I20240501 20:09:23.989256 11591 raft_consensus.cc:348] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } }
I20240501 20:09:23.989560 11591 raft_consensus.cc:374] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.989598 11593 raft_consensus.cc:374] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:23.989670 11591 raft_consensus.cc:729] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dc38b7a43240e68f3d9b7fe6c40191, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.989706 11593 raft_consensus.cc:729] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb64f4d9d8f4232a3beec44e45ccb3a, State: Initialized, Role: FOLLOWER
I20240501 20:09:23.990064 11592 consensus_queue.cc:260] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } }
I20240501 20:09:23.990339 11591 consensus_queue.cc:260] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } }
I20240501 20:09:23.990485 11593 consensus_queue.cc:260] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } }
I20240501 20:09:23.993054 11443 heartbeater.cc:503] Master 127.8.68.254:41155 was elected leader, sending a full tablet report...
I20240501 20:09:23.993705 11592 ts_tablet_manager.cc:1398] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20240501 20:09:23.995363 11313 heartbeater.cc:503] Master 127.8.68.254:41155 was elected leader, sending a full tablet report...
I20240501 20:09:23.995800 11591 ts_tablet_manager.cc:1398] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.007s
I20240501 20:09:23.997164 11593 ts_tablet_manager.cc:1398] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.008s
I20240501 20:09:23.998653 11598 raft_consensus.cc:482] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:23.998858 11598 raft_consensus.cc:504] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } }
I20240501 20:09:24.000838 11598 leader_election.cc:288] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6058e29fedba4c69a8725ad7f505e378 (127.8.68.196:38399), 2eb64f4d9d8f4232a3beec44e45ccb3a (127.8.68.193:38105)
I20240501 20:09:24.011677 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241afb3dadd747e991a08b17f5d7a581" candidate_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6058e29fedba4c69a8725ad7f505e378" is_pre_election: true
I20240501 20:09:24.012013 11398 raft_consensus.cc:2454] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58dc38b7a43240e68f3d9b7fe6c40191 in term 0.
I20240501 20:09:24.012523 11202 leader_election.cc:302] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58dc38b7a43240e68f3d9b7fe6c40191, 6058e29fedba4c69a8725ad7f505e378; no voters: 
I20240501 20:09:24.012848 11598 raft_consensus.cc:2781] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:24.012950 11598 raft_consensus.cc:482] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:24.013038 11598 raft_consensus.cc:3037] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.013012 11008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241afb3dadd747e991a08b17f5d7a581" candidate_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" is_pre_election: true
I20240501 20:09:24.013386 11008 raft_consensus.cc:2454] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58dc38b7a43240e68f3d9b7fe6c40191 in term 0.
I20240501 20:09:24.015133 11598 raft_consensus.cc:504] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } }
I20240501 20:09:24.015674 11598 leader_election.cc:288] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [CANDIDATE]: Term 1 election: Requested vote from peers 6058e29fedba4c69a8725ad7f505e378 (127.8.68.196:38399), 2eb64f4d9d8f4232a3beec44e45ccb3a (127.8.68.193:38105)
I20240501 20:09:24.016002 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241afb3dadd747e991a08b17f5d7a581" candidate_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6058e29fedba4c69a8725ad7f505e378"
I20240501 20:09:24.016166 11398 raft_consensus.cc:3037] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.016152 11008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241afb3dadd747e991a08b17f5d7a581" candidate_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a"
I20240501 20:09:24.016309 11008 raft_consensus.cc:3037] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:24.018086 11398 raft_consensus.cc:2454] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58dc38b7a43240e68f3d9b7fe6c40191 in term 1.
I20240501 20:09:24.018272 11008 raft_consensus.cc:2454] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58dc38b7a43240e68f3d9b7fe6c40191 in term 1.
I20240501 20:09:24.018405 11202 leader_election.cc:302] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58dc38b7a43240e68f3d9b7fe6c40191, 6058e29fedba4c69a8725ad7f505e378; no voters: 
I20240501 20:09:24.018661 11598 raft_consensus.cc:2781] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:24.019045 11598 raft_consensus.cc:686] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [term 1 LEADER]: Becoming Leader. State: Replica: 58dc38b7a43240e68f3d9b7fe6c40191, State: Running, Role: LEADER
I20240501 20:09:24.019451 11598 consensus_queue.cc:237] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } }
I20240501 20:09:24.022998 10866 catalog_manager.cc:5484] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58dc38b7a43240e68f3d9b7fe6c40191 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dc38b7a43240e68f3d9b7fe6c40191" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 41409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:24.044270  8467 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:09:24.046373  8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2eb64f4d9d8f4232a3beec44e45ccb3a to finish bootstrapping
I20240501 20:09:24.053759  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 05a79835c84349a0b37cd7ce1c5fa1fa to finish bootstrapping
W20240501 20:09:24.058347 11314 tablet.cc:2369] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:24.062224  8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 58dc38b7a43240e68f3d9b7fe6c40191 to finish bootstrapping
I20240501 20:09:24.069118  8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6058e29fedba4c69a8725ad7f505e378 to finish bootstrapping
I20240501 20:09:24.075673  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 87ff7e2623c84f29aa73f663e72ca1f1 to finish bootstrapping
W20240501 20:09:24.219616 11054 tablet.cc:2369] T 241afb3dadd747e991a08b17f5d7a581 P 2eb64f4d9d8f4232a3beec44e45ccb3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:09:24.231061 11444 tablet.cc:2369] T 241afb3dadd747e991a08b17f5d7a581 P 6058e29fedba4c69a8725ad7f505e378: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:24.497581 11118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:24.507440 11378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:24.508179 11248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:24.508414 10988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:24.511545 11508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:24.516206 11598 consensus_queue.cc:1035] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6058e29fedba4c69a8725ad7f505e378" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:24.573473 11639 consensus_queue.cc:1035] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb64f4d9d8f4232a3beec44e45ccb3a" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:24.669271 11118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:24.670560 11248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:24.671963 10988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:24.679569 11508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:24.686236 11378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:24.904297 11573 heartbeater.cc:503] Master 127.8.68.254:41155 was elected leader, sending a full tablet report...
I20240501 20:09:25.269289 10988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.275673 11118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.277706 11378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.288132 11248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:25.295969 11508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.423866 11118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.435986 11248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:25.436465 11378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.436863 10988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.443943 11508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.601171 11118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.604557 11508 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.606200 10988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.607781 11378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:25.608405 11248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:25.758869  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 10926
I20240501 20:09:25.760175 11048 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:25.851006  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 10926
I20240501 20:09:25.866466  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 11056
I20240501 20:09:25.867545 11178 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:25.953233  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 11056
I20240501 20:09:25.967861  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 11186
I20240501 20:09:25.968732 11308 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:09:26.061595 11202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:38105: connect: Connection refused (error 111)
W20240501 20:09:26.063088 11202 consensus_peers.cc:485] T 241afb3dadd747e991a08b17f5d7a581 P 58dc38b7a43240e68f3d9b7fe6c40191 -> Peer 2eb64f4d9d8f4232a3beec44e45ccb3a (127.8.68.193:38105): Couldn't send request to peer 2eb64f4d9d8f4232a3beec44e45ccb3a. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:38105: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:26.066637  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 11186
I20240501 20:09:26.081368  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 11316
I20240501 20:09:26.082383 11438 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:26.173310  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 11316
I20240501 20:09:26.187588  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 11446
I20240501 20:09:26.188639 11567 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:26.274531  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 11446
I20240501 20:09:26.290315  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 10836
I20240501 20:09:26.291379 10897 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:26.399294  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 10836
2024-05-01T20:09:26Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (5164 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5164 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/3
I20240501 20:09:26.436422  8467 test_util.cc:268] Using random seed: 272200239
I20240501 20:09:26.438196  8467 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:09:26.438261  8467 ts_itest-base.cc:113] --------------
I20240501 20:09:26.438347  8467 ts_itest-base.cc:114] 7 tablet servers
I20240501 20:09:26.438390  8467 ts_itest-base.cc:115] 2 replicas per TS
I20240501 20:09:26.438460  8467 ts_itest-base.cc:116] --------------
2024-05-01T20:09:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:09:26Z Disabled control of system clock
I20240501 20:09:26.467327  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:46165
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:46165
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 20:09:26.763859 11801 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:09:26.764158 11801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:26.764254 11801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:26.764349 11801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:26.771346 11801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:26.771453 11801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:09:26.771545 11801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:26.771605 11801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:09:26.771684 11801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:09:26.780067 11801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:46165
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:26.781129 11801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:26.782725 11801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:26.792551 11807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:26.793042 11806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:26.793454 11809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:26.794039 11801 server_base.cc:1034] running on GCE node
I20240501 20:09:26.794734 11801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:26.795894 11801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:26.797106 11801 hybrid_clock.cc:648] HybridClock initialized: now 1714594166797052 us; error 32 us; skew 500 ppm
I20240501 20:09:26.797389 11801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:26.799365 11801 webserver.cc:460] Webserver started at http://127.8.68.254:43097/ using document root <none> and password file <none>
I20240501 20:09:26.799976 11801 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:26.800102 11801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:26.800372 11801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:26.802759 11801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "466d4986891e4af2a4e44757733bfd5c"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-08p9"
I20240501 20:09:26.803416 11801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "466d4986891e4af2a4e44757733bfd5c"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-08p9"
I20240501 20:09:26.807933 11801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:09:26.810974 11815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:26.812105 11801 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:09:26.812256 11801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "466d4986891e4af2a4e44757733bfd5c"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-08p9"
I20240501 20:09:26.812433 11801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:26.828090 11801 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:26.828727 11801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:26.828974 11801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:26.844727 11801 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:46165
I20240501 20:09:26.844766 11866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:46165 every 8 connection(s)
I20240501 20:09:26.845893 11801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:09:26.849156 11867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.853053  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 11801
I20240501 20:09:26.853228  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:09:26.857851 11867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c: Bootstrap starting.
I20240501 20:09:26.860831 11867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.861987 11867 log.cc:824] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:26.864586 11867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c: No bootstrap required, opened a new log
I20240501 20:09:26.870340 11867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466d4986891e4af2a4e44757733bfd5c" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46165 } }
I20240501 20:09:26.870620 11867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.870698 11867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 466d4986891e4af2a4e44757733bfd5c, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.871268 11867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466d4986891e4af2a4e44757733bfd5c" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46165 } }
I20240501 20:09:26.871487 11867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:26.871613 11867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:26.871759 11867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:26.873591 11867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466d4986891e4af2a4e44757733bfd5c" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46165 } }
I20240501 20:09:26.874009 11867 leader_election.cc:302] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 466d4986891e4af2a4e44757733bfd5c; no voters: 
I20240501 20:09:26.874516 11867 leader_election.cc:288] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:09:26.874606 11872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:26.875155 11872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [term 1 LEADER]: Becoming Leader. State: Replica: 466d4986891e4af2a4e44757733bfd5c, State: Running, Role: LEADER
I20240501 20:09:26.875622 11872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466d4986891e4af2a4e44757733bfd5c" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46165 } }
I20240501 20:09:26.876259 11867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:09:26.878422 11874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 466d4986891e4af2a4e44757733bfd5c. Latest consensus state: current_term: 1 leader_uuid: "466d4986891e4af2a4e44757733bfd5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466d4986891e4af2a4e44757733bfd5c" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46165 } } }
I20240501 20:09:26.878741 11874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:26.878587 11873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "466d4986891e4af2a4e44757733bfd5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466d4986891e4af2a4e44757733bfd5c" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46165 } } }
I20240501 20:09:26.878846 11873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:26.879990 11879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:09:26.883716 11879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:09:26.890985 11879 catalog_manager.cc:1347] Generated new cluster ID: 88388ba23c76427c9d3b3f37340c49a3
I20240501 20:09:26.891081 11879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:09:26.912897 11879 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:09:26.914382 11879 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:09:26.923528 11879 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 466d4986891e4af2a4e44757733bfd5c: Generated new TSK 0
I20240501 20:09:26.924319 11879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:09:26.929037  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:09:27.233831 11891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:27.234077 11891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:27.234174 11891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:27.241261 11891 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:27.241353 11891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:27.241464 11891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240501 20:09:27.249806 11891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:27.250921 11891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:27.252455 11891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:27.263342 11896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:27.263851 11899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:27.264281 11897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:27.264775 11891 server_base.cc:1034] running on GCE node
I20240501 20:09:27.265271 11891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:27.266045 11891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:27.267235 11891 hybrid_clock.cc:648] HybridClock initialized: now 1714594167267193 us; error 34 us; skew 500 ppm
I20240501 20:09:27.267503 11891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:27.269400 11891 webserver.cc:460] Webserver started at http://127.8.68.193:38815/ using document root <none> and password file <none>
I20240501 20:09:27.270031 11891 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:27.270136 11891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:27.270442 11891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:27.272711 11891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "dc71b180f72b4c4697d28b13b363d705"
format_stamp: "Formatted at 2024-05-01 20:09:27 on dist-test-slave-08p9"
I20240501 20:09:27.273394 11891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "dc71b180f72b4c4697d28b13b363d705"
format_stamp: "Formatted at 2024-05-01 20:09:27 on dist-test-slave-08p9"
I20240501 20:09:27.277905 11891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20240501 20:09:27.280866 11905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:27.281948 11891 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:27.282096 11891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "dc71b180f72b4c4697d28b13b363d705"
format_stamp: "Formatted at 2024-05-01 20:09:27 on dist-test-slave-08p9"
I20240501 20:09:27.282261 11891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:27.300321 11891 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:27.300978 11891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:27.301216 11891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:27.301985 11891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:27.303423 11891 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:27.303539 11891 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:27.303637 11891 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:27.303704 11891 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:27.332249 11891 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:39397
I20240501 20:09:27.332309 12017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:39397 every 8 connection(s)
I20240501 20:09:27.333384 11891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:09:27.337922  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 11891
I20240501 20:09:27.338109  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:09:27.341187  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:27.345320 12018 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:27.345585 12018 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:27.346307 12018 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:27.348255 11832 ts_manager.cc:194] Registered new tserver with Master: dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:27.349668 11832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:44797
W20240501 20:09:27.641072 12021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:27.641310 12021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:27.641438 12021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:27.648380 12021 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:27.648494 12021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:27.648627 12021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240501 20:09:27.656790 12021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:27.657814 12021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:27.659351 12021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:27.670066 12026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:27.670585 12027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:27.671013 12029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:27.671505 12021 server_base.cc:1034] running on GCE node
I20240501 20:09:27.671983 12021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:27.672724 12021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:27.673895 12021 hybrid_clock.cc:648] HybridClock initialized: now 1714594167673848 us; error 45 us; skew 500 ppm
I20240501 20:09:27.674166 12021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:27.676059 12021 webserver.cc:460] Webserver started at http://127.8.68.194:37699/ using document root <none> and password file <none>
I20240501 20:09:27.676710 12021 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:27.676836 12021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:27.677162 12021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:27.679464 12021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "34f8af7f2d4245899a7d8fd5012a3721"
format_stamp: "Formatted at 2024-05-01 20:09:27 on dist-test-slave-08p9"
I20240501 20:09:27.680193 12021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "34f8af7f2d4245899a7d8fd5012a3721"
format_stamp: "Formatted at 2024-05-01 20:09:27 on dist-test-slave-08p9"
I20240501 20:09:27.684572 12021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.003s
I20240501 20:09:27.687680 12035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:27.688859 12021 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:27.688993 12021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "34f8af7f2d4245899a7d8fd5012a3721"
format_stamp: "Formatted at 2024-05-01 20:09:27 on dist-test-slave-08p9"
I20240501 20:09:27.689157 12021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:27.718123 12021 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:27.718797 12021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:27.719031 12021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:27.719796 12021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:27.721122 12021 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:27.721206 12021 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:27.721310 12021 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:27.721376 12021 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:27.748562 12021 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:32853
I20240501 20:09:27.748632 12147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:32853 every 8 connection(s)
I20240501 20:09:27.749640 12021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:09:27.751199  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 12021
I20240501 20:09:27.751360  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:09:27.754516  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:27.760946 12148 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:27.761204 12148 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:27.761973 12148 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:27.763288 11832 ts_manager.cc:194] Registered new tserver with Master: 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:27.763981 11832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:36593
W20240501 20:09:28.052464 12151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:28.052695 12151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:28.052822 12151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:28.059631 12151 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:28.059743 12151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:28.059885 12151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240501 20:09:28.068153 12151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:28.069155 12151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:28.070668 12151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:28.081306 12156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:28.081869 12157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:28.082353 12159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:28.082872 12151 server_base.cc:1034] running on GCE node
I20240501 20:09:28.083432 12151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:28.084291 12151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:28.085508 12151 hybrid_clock.cc:648] HybridClock initialized: now 1714594168085461 us; error 38 us; skew 500 ppm
I20240501 20:09:28.085810 12151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:28.087947 12151 webserver.cc:460] Webserver started at http://127.8.68.195:34387/ using document root <none> and password file <none>
I20240501 20:09:28.088608 12151 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:28.088724 12151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:28.089041 12151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:28.091493 12151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "18717c9754d04918815985cf07c1ddad"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:28.092243 12151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "18717c9754d04918815985cf07c1ddad"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:28.096920 12151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20240501 20:09:28.100036 12165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:28.101171 12151 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:28.101320 12151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "18717c9754d04918815985cf07c1ddad"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:28.101501 12151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:28.133352 12151 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:28.134085 12151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:28.134377 12151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:28.135252 12151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:28.136658 12151 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:28.136765 12151 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:28.136875 12151 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:28.136957 12151 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:28.166857 12151 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:42201
I20240501 20:09:28.166927 12277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:42201 every 8 connection(s)
I20240501 20:09:28.168006 12151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:09:28.174384  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 12151
I20240501 20:09:28.174566  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:09:28.177891  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:28.179967 12278 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:28.180215 12278 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:28.180949 12278 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:28.182199 11832 ts_manager.cc:194] Registered new tserver with Master: 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:28.182967 11832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:42973
I20240501 20:09:28.352483 12018 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
W20240501 20:09:28.483194 12281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:28.483426 12281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:28.483561 12281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:28.490597 12281 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:28.490697 12281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:28.490839 12281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:09:28.499086 12281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:28.500109 12281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:28.501674 12281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:28.512792 12286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:28.513280 12289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:28.513553 12287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:28.514084 12281 server_base.cc:1034] running on GCE node
I20240501 20:09:28.514585 12281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:28.515375 12281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:28.516546 12281 hybrid_clock.cc:648] HybridClock initialized: now 1714594168516499 us; error 44 us; skew 500 ppm
I20240501 20:09:28.516819 12281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:28.518826 12281 webserver.cc:460] Webserver started at http://127.8.68.196:41693/ using document root <none> and password file <none>
I20240501 20:09:28.519439 12281 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:28.519541 12281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:28.519829 12281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:28.522046 12281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3b95f8c6589e473bb21db7681e227638"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:28.522763 12281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3b95f8c6589e473bb21db7681e227638"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:28.527154 12281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:09:28.530015 12295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:28.531169 12281 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20240501 20:09:28.531299 12281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3b95f8c6589e473bb21db7681e227638"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:28.531459 12281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:28.560310 12281 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:28.560951 12281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:28.561189 12281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:28.561959 12281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:28.563251 12281 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:28.563342 12281 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:28.563438 12281 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:28.563511 12281 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:28.591243 12281 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:36351
I20240501 20:09:28.591320 12407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:36351 every 8 connection(s)
I20240501 20:09:28.592283 12281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:09:28.597345  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 12281
I20240501 20:09:28.597528  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:09:28.600796  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:28.604619 12408 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:28.604935 12408 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:28.605898 12408 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:28.607302 11832 ts_manager.cc:194] Registered new tserver with Master: 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351)
I20240501 20:09:28.608095 11832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:47461
I20240501 20:09:28.766500 12148 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
W20240501 20:09:28.900725 12411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:28.900957 12411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:28.901046 12411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:28.908377 12411 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:28.908473 12411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:28.908604 12411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:09:28.916754 12411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:28.917797 12411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:28.919332 12411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:28.929863 12416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:28.930393 12417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:28.930718 12419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:28.931402 12411 server_base.cc:1034] running on GCE node
I20240501 20:09:28.931888 12411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:28.932657 12411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:28.933840 12411 hybrid_clock.cc:648] HybridClock initialized: now 1714594168933800 us; error 32 us; skew 500 ppm
I20240501 20:09:28.934113 12411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:28.936120 12411 webserver.cc:460] Webserver started at http://127.8.68.197:39737/ using document root <none> and password file <none>
I20240501 20:09:28.936779 12411 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:28.936878 12411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:28.937150 12411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:28.939513 12411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "535367472879403fa999dd00a155f042"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:28.940238 12411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "535367472879403fa999dd00a155f042"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:28.944666 12411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 20:09:28.947603 12425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:28.948762 12411 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:28.948912 12411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "535367472879403fa999dd00a155f042"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:28.949103 12411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:28.968209 12411 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:28.968921 12411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:28.969187 12411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:28.970038 12411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:28.971453 12411 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:28.971542 12411 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:28.971625 12411 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:28.971683 12411 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:29.001183 12411 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:43403
I20240501 20:09:29.001226 12537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:43403 every 8 connection(s)
I20240501 20:09:29.002380 12411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:09:29.010869  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 12411
I20240501 20:09:29.011065  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:09:29.013739 12538 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:29.014019 12538 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:29.014448  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:0
--local_ip_for_outbound_sockets=127.8.68.198
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:29.014914 12538 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:29.016311 11831 ts_manager.cc:194] Registered new tserver with Master: 535367472879403fa999dd00a155f042 (127.8.68.197:43403)
I20240501 20:09:29.017221 11831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:47923
I20240501 20:09:29.185189 12278 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
W20240501 20:09:29.320120 12541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:29.320369 12541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:29.320502 12541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:29.327446 12541 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:29.327562 12541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:29.327698 12541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240501 20:09:29.337391 12541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:29.338517 12541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:29.340073 12541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:29.350909 12546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:29.351470 12547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:29.351891 12549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:29.352461 12541 server_base.cc:1034] running on GCE node
I20240501 20:09:29.353014 12541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:29.353799 12541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:29.355006 12541 hybrid_clock.cc:648] HybridClock initialized: now 1714594169354956 us; error 45 us; skew 500 ppm
I20240501 20:09:29.355297 12541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:29.357263 12541 webserver.cc:460] Webserver started at http://127.8.68.198:39883/ using document root <none> and password file <none>
I20240501 20:09:29.357892 12541 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:29.357995 12541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:29.358307 12541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:29.360800 12541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "463bd46a1f6e4e89a3040c4cf50dd919"
format_stamp: "Formatted at 2024-05-01 20:09:29 on dist-test-slave-08p9"
I20240501 20:09:29.361639 12541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "463bd46a1f6e4e89a3040c4cf50dd919"
format_stamp: "Formatted at 2024-05-01 20:09:29 on dist-test-slave-08p9"
I20240501 20:09:29.366842 12541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.003s
I20240501 20:09:29.369848 12555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:29.371033 12541 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:09:29.371171 12541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "463bd46a1f6e4e89a3040c4cf50dd919"
format_stamp: "Formatted at 2024-05-01 20:09:29 on dist-test-slave-08p9"
I20240501 20:09:29.371341 12541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:29.416175 12541 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:29.416898 12541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:29.417142 12541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:29.417943 12541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:29.419241 12541 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:29.419332 12541 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:29.419435 12541 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:29.419503 12541 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:29.447352 12541 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:35531
I20240501 20:09:29.447432 12667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:35531 every 8 connection(s)
I20240501 20:09:29.448465 12541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:09:29.454475  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 12541
I20240501 20:09:29.454668  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:09:29.458055  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:0
--local_ip_for_outbound_sockets=127.8.68.199
--webserver_interface=127.8.68.199
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:29.460062 12668 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:29.460300 12668 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:29.460956 12668 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:29.462168 11831 ts_manager.cc:194] Registered new tserver with Master: 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198:35531)
I20240501 20:09:29.462988 11831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:57547
I20240501 20:09:29.610314 12408 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
W20240501 20:09:29.760322 12671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:29.760546 12671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:29.760636 12671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:29.767704 12671 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:29.767797 12671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:29.767903 12671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240501 20:09:29.776381 12671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:29.777405 12671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:29.778923 12671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:29.789489 12676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:29.790036 12677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:29.790501 12679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:29.791020 12671 server_base.cc:1034] running on GCE node
I20240501 20:09:29.791460 12671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:29.792200 12671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:29.793406 12671 hybrid_clock.cc:648] HybridClock initialized: now 1714594169793334 us; error 44 us; skew 500 ppm
I20240501 20:09:29.793664 12671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:29.795578 12671 webserver.cc:460] Webserver started at http://127.8.68.199:42839/ using document root <none> and password file <none>
I20240501 20:09:29.796195 12671 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:29.796280 12671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:29.796510 12671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:29.798702 12671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "aed522b44a8e420daf65718450541c3c"
format_stamp: "Formatted at 2024-05-01 20:09:29 on dist-test-slave-08p9"
I20240501 20:09:29.799343 12671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "aed522b44a8e420daf65718450541c3c"
format_stamp: "Formatted at 2024-05-01 20:09:29 on dist-test-slave-08p9"
I20240501 20:09:29.803642 12671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:09:29.806556 12685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:29.807760 12671 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:09:29.807893 12671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "aed522b44a8e420daf65718450541c3c"
format_stamp: "Formatted at 2024-05-01 20:09:29 on dist-test-slave-08p9"
I20240501 20:09:29.808053 12671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:29.822253 12671 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:29.822937 12671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:29.823166 12671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:29.823931 12671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:29.825155 12671 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:29.825242 12671 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:29.825340 12671 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:29.825403 12671 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:29.852348 12671 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:40105
I20240501 20:09:29.852380 12797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:40105 every 8 connection(s)
I20240501 20:09:29.853761 12671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:09:29.856664  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 12671
I20240501 20:09:29.856877  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:09:29.864964 12798 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:29.865190 12798 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:29.865845 12798 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:29.866978 11831 ts_manager.cc:194] Registered new tserver with Master: aed522b44a8e420daf65718450541c3c (127.8.68.199:40105)
I20240501 20:09:29.867687 11831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:55537
I20240501 20:09:29.872100  8467 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:09:29.885720 11831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49824:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240501 20:09:29.923647 12083 tablet_service.cc:1467] Processing CreateTablet for tablet 262853fe9d5c4aa7acdac809a6378567 (DEFAULT_TABLE table=TestTable [id=6791900190fd42098c6b5729056d1534]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.923647 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 262853fe9d5c4aa7acdac809a6378567 (DEFAULT_TABLE table=TestTable [id=6791900190fd42098c6b5729056d1534]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.925066 11953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262853fe9d5c4aa7acdac809a6378567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.925078 12083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262853fe9d5c4aa7acdac809a6378567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.937528 12814 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:29.938669 12815 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:29.940932 12814 tablet_bootstrap.cc:654] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.941848 12815 tablet_bootstrap.cc:654] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.942140 12814 log.cc:824] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:29.942968 12815 log.cc:824] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:29.944635 12814 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:29.944890 12814 ts_tablet_manager.cc:1367] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240501 20:09:29.945397 12815 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:29.945652 12815 ts_tablet_manager.cc:1367] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.005s
I20240501 20:09:29.953078 12814 raft_consensus.cc:348] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:29.953444 12814 raft_consensus.cc:374] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.953552 12814 raft_consensus.cc:729] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.953724 12815 raft_consensus.cc:348] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:29.954062 12815 raft_consensus.cc:374] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.954167 12815 raft_consensus.cc:729] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.954193 12814 consensus_queue.cc:260] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:29.954912 12815 consensus_queue.cc:260] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:29.955689 12814 ts_tablet_manager.cc:1398] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.011s	user 0.012s	sys 0.000s
I20240501 20:09:29.956262 12815 ts_tablet_manager.cc:1398] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.001s
W20240501 20:09:30.002346 12149 tablet.cc:2369] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:30.019805 12538 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
I20240501 20:09:30.052125 12819 raft_consensus.cc:482] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:30.052328 12819 raft_consensus.cc:504] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:30.053669 12819 leader_election.cc:288] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:30.059520 12103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262853fe9d5c4aa7acdac809a6378567" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721" is_pre_election: true
I20240501 20:09:30.059911 12103 raft_consensus.cc:2454] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 0.
I20240501 20:09:30.060444 11908 leader_election.cc:302] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:30.060761 12819 raft_consensus.cc:2781] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:30.060895 12819 raft_consensus.cc:482] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:30.060992 12819 raft_consensus.cc:3037] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.062891 12819 raft_consensus.cc:504] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:30.063289 12819 leader_election.cc:288] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 election: Requested vote from peers 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:30.063709 12103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262853fe9d5c4aa7acdac809a6378567" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
I20240501 20:09:30.063848 12103 raft_consensus.cc:3037] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.065704 12103 raft_consensus.cc:2454] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 1.
I20240501 20:09:30.065995 11908 leader_election.cc:302] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:30.066221 12819 raft_consensus.cc:2781] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:30.066607 12819 raft_consensus.cc:686] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Becoming Leader. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Running, Role: LEADER
I20240501 20:09:30.067031 12819 consensus_queue.cc:237] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:30.071484 11832 catalog_manager.cc:5484] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc71b180f72b4c4697d28b13b363d705 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:30.086407 12019 tablet.cc:2369] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:30.102772  8467 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:09:30.104928  8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dc71b180f72b4c4697d28b13b363d705 to finish bootstrapping
I20240501 20:09:30.112586  8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 34f8af7f2d4245899a7d8fd5012a3721 to finish bootstrapping
I20240501 20:09:30.118983  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 18717c9754d04918815985cf07c1ddad to finish bootstrapping
I20240501 20:09:30.126202  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3b95f8c6589e473bb21db7681e227638 to finish bootstrapping
I20240501 20:09:30.133515  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 535367472879403fa999dd00a155f042 to finish bootstrapping
I20240501 20:09:30.140934  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 463bd46a1f6e4e89a3040c4cf50dd919 to finish bootstrapping
I20240501 20:09:30.147714  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver aed522b44a8e420daf65718450541c3c to finish bootstrapping
I20240501 20:09:30.155050  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 12281
I20240501 20:09:30.155779 12403 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:30.232921  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 12281
I20240501 20:09:30.245714  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 12411
I20240501 20:09:30.246673 12533 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:30.325366  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 12411
I20240501 20:09:30.338047  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 12541
I20240501 20:09:30.339282 12663 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:30.416013  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 12541
I20240501 20:09:30.428225  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 12671
I20240501 20:09:30.429198 12793 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:30.506299  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 12671
I20240501 20:09:30.660102 12834 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:34.270114 11831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49824:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:34.280035 12083 tablet_service.cc:1467] Processing CreateTablet for tablet cf6c676aa51e44a9b8a1c2a3c828ccd0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=801bc83cb7614cdaaf85134efd64f1d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.280498 12083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6c676aa51e44a9b8a1c2a3c828ccd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.280556 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad26a211a9c4c609da559d2681c837a (DEFAULT_TABLE table=rebalance_test_table_0 [id=801bc83cb7614cdaaf85134efd64f1d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.280555 11952 tablet_service.cc:1467] Processing CreateTablet for tablet 2c0a1865e31241569216a3b4b072829c (DEFAULT_TABLE table=rebalance_test_table_0 [id=801bc83cb7614cdaaf85134efd64f1d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.281029 11953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad26a211a9c4c609da559d2681c837a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.281275 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0a1865e31241569216a3b4b072829c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.292711 12212 tablet_service.cc:1467] Processing CreateTablet for tablet cf6c676aa51e44a9b8a1c2a3c828ccd0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=801bc83cb7614cdaaf85134efd64f1d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.293386 12211 tablet_service.cc:1467] Processing CreateTablet for tablet 2c0a1865e31241569216a3b4b072829c (DEFAULT_TABLE table=rebalance_test_table_0 [id=801bc83cb7614cdaaf85134efd64f1d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.294289 12211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0a1865e31241569216a3b4b072829c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.294746 12212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6c676aa51e44a9b8a1c2a3c828ccd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.295930 12850 tablet_bootstrap.cc:492] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:34.292845 12213 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad26a211a9c4c609da559d2681c837a (DEFAULT_TABLE table=rebalance_test_table_0 [id=801bc83cb7614cdaaf85134efd64f1d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.296703 12213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad26a211a9c4c609da559d2681c837a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.297621 12850 tablet_bootstrap.cc:654] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.299270 12851 tablet_bootstrap.cc:492] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:34.301716 12851 tablet_bootstrap.cc:654] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.305471 12850 tablet_bootstrap.cc:492] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:34.305649 12850 ts_tablet_manager.cc:1367] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.005s
I20240501 20:09:34.306511 12850 raft_consensus.cc:348] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:34.306775 12850 raft_consensus.cc:374] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.306871 12850 raft_consensus.cc:729] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.307134 12850 consensus_queue.cc:260] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:34.312302 12855 tablet_bootstrap.cc:492] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:34.313066 12851 tablet_bootstrap.cc:492] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:34.313181 12851 ts_tablet_manager.cc:1367] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.005s
I20240501 20:09:34.313956 12850 ts_tablet_manager.cc:1398] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20240501 20:09:34.313954 12851 raft_consensus.cc:348] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:34.314127 12851 raft_consensus.cc:374] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.314216 12851 raft_consensus.cc:729] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.314457 12851 consensus_queue.cc:260] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:34.314991 12851 ts_tablet_manager.cc:1398] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:34.315251 12851 tablet_bootstrap.cc:492] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:34.316015 12855 tablet_bootstrap.cc:654] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.316618 12851 tablet_bootstrap.cc:654] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.317196 12855 log.cc:824] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:34.319527 12851 tablet_bootstrap.cc:492] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:34.319655 12851 ts_tablet_manager.cc:1367] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:09:34.319756 12855 tablet_bootstrap.cc:492] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:34.319972 12855 ts_tablet_manager.cc:1367] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.005s
I20240501 20:09:34.320204 12851 raft_consensus.cc:348] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:34.320362 12851 raft_consensus.cc:374] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.320425 12851 raft_consensus.cc:729] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.320613 12851 consensus_queue.cc:260] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:34.320982 12851 ts_tablet_manager.cc:1398] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:09:34.326316 12855 raft_consensus.cc:348] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:34.326675 12855 raft_consensus.cc:374] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.326766 12855 raft_consensus.cc:729] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.327297 12855 consensus_queue.cc:260] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:34.328658 12855 ts_tablet_manager.cc:1398] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.005s
I20240501 20:09:34.329030 12855 tablet_bootstrap.cc:492] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:34.330082 12855 tablet_bootstrap.cc:654] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.331853 12855 tablet_bootstrap.cc:492] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:34.331991 12855 ts_tablet_manager.cc:1367] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:34.332677 12855 raft_consensus.cc:348] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:34.332805 12855 raft_consensus.cc:374] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.332888 12855 raft_consensus.cc:729] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.333094 12855 consensus_queue.cc:260] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:34.333417 12855 ts_tablet_manager.cc:1398] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:09:34.333657 12855 tablet_bootstrap.cc:492] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:34.334934 12855 tablet_bootstrap.cc:654] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.338552 12855 tablet_bootstrap.cc:492] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:34.338677 12855 ts_tablet_manager.cc:1367] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 20:09:34.339131 12855 raft_consensus.cc:348] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:34.339268 12855 raft_consensus.cc:374] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.339334 12855 raft_consensus.cc:729] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.339521 12855 consensus_queue.cc:260] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:34.339924 12855 ts_tablet_manager.cc:1398] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240501 20:09:34.363934 12843 raft_consensus.cc:482] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:34.364066 12843 raft_consensus.cc:504] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:34.364639 12843 leader_election.cc:288] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:34.370539 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0a1865e31241569216a3b4b072829c" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad" is_pre_election: true
I20240501 20:09:34.371024 12233 raft_consensus.cc:2454] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 0.
I20240501 20:09:34.371515 11906 leader_election.cc:302] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:34.371778 12843 raft_consensus.cc:2781] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:34.371891 12843 raft_consensus.cc:482] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:34.372001 12843 raft_consensus.cc:3037] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.373847 12843 raft_consensus.cc:504] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:34.374311 12843 leader_election.cc:288] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 election: Requested vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:34.374572 12854 raft_consensus.cc:482] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:34.374784 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0a1865e31241569216a3b4b072829c" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad"
I20240501 20:09:34.374753 12854 raft_consensus.cc:504] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:34.374961 12233 raft_consensus.cc:3037] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.376435 12854 leader_election.cc:288] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:34.377641 12233 raft_consensus.cc:2454] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 1.
I20240501 20:09:34.378000 11906 leader_election.cc:302] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:34.378294 12843 raft_consensus.cc:2781] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:34.378686 12843 raft_consensus.cc:686] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Becoming Leader. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Running, Role: LEADER
I20240501 20:09:34.378979 12843 consensus_queue.cc:237] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:34.381124 11831 catalog_manager.cc:5484] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc71b180f72b4c4697d28b13b363d705 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:34.383944 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6c676aa51e44a9b8a1c2a3c828ccd0" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad" is_pre_election: true
I20240501 20:09:34.384186 12233 raft_consensus.cc:2454] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 0.
I20240501 20:09:34.384570 12036 leader_election.cc:302] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:34.384922 12854 raft_consensus.cc:2781] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:34.385043 12854 raft_consensus.cc:482] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:34.385138 12854 raft_consensus.cc:3037] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.387099 12854 raft_consensus.cc:504] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:34.387506 12854 leader_election.cc:288] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Requested vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:34.387894 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6c676aa51e44a9b8a1c2a3c828ccd0" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad"
I20240501 20:09:34.388039 12233 raft_consensus.cc:3037] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.389714 12233 raft_consensus.cc:2454] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 1.
I20240501 20:09:34.390012 12036 leader_election.cc:302] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:34.390251 12854 raft_consensus.cc:2781] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:34.390640 12854 raft_consensus.cc:686] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:34.390995 12854 consensus_queue.cc:237] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:34.394134 11831 catalog_manager.cc:5484] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } }
W20240501 20:09:34.423142 12279 tablet.cc:2369] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:34.587018 12858 raft_consensus.cc:482] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:34.587219 12858 raft_consensus.cc:504] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:34.588591 12858 leader_election.cc:288] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:34.595407 11973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad26a211a9c4c609da559d2681c837a" candidate_uuid: "18717c9754d04918815985cf07c1ddad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705" is_pre_election: true
I20240501 20:09:34.595810 11973 raft_consensus.cc:2454] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18717c9754d04918815985cf07c1ddad in term 0.
I20240501 20:09:34.596341 12166 leader_election.cc:302] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:34.596668 12858 raft_consensus.cc:2781] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:34.596786 12858 raft_consensus.cc:482] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:34.596871 12858 raft_consensus.cc:3037] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.599201 12858 raft_consensus.cc:504] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:34.599594 12858 leader_election.cc:288] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:34.600040 11973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad26a211a9c4c609da559d2681c837a" candidate_uuid: "18717c9754d04918815985cf07c1ddad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:34.600211 11973 raft_consensus.cc:3037] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.602097 11973 raft_consensus.cc:2454] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18717c9754d04918815985cf07c1ddad in term 1.
I20240501 20:09:34.602439 12166 leader_election.cc:302] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:34.602715 12858 raft_consensus.cc:2781] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:34.603142 12858 raft_consensus.cc:686] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Becoming Leader. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Running, Role: LEADER
I20240501 20:09:34.603583 12858 consensus_queue.cc:237] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:34.607781 11831 catalog_manager.cc:5484] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad reported cstate change: term changed from 0 to 1, leader changed from <none> to 18717c9754d04918815985cf07c1ddad (127.8.68.195). New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:34.945693 12871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:34.945946 12871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:34.952769 12871 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:34.970031 12843 consensus_queue.cc:1035] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 20:09:34.996749 12897 consensus_queue.cc:1035] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:35.090554 12858 consensus_queue.cc:1035] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:35.111810 12885 mvcc.cc:205] Tried to move back new op lower bound from 7022977741219241984 to 7022977739179192320. Current Snapshot: MvccSnapshot[applied={T|T < 7022977741219241984}]

INSERT report
    rows total: 2000
    time total: 153.931 ms
  time per row: 0.0769654 ms
I20240501 20:09:35.246866 11830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49824:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:35.254797 12083 tablet_service.cc:1467] Processing CreateTablet for tablet 9d7dd13822784e789d558b4cc6c5010e (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.254889 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 9d7dd13822784e789d558b4cc6c5010e (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.255267 12083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d7dd13822784e789d558b4cc6c5010e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.255302 11953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d7dd13822784e789d558b4cc6c5010e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.255477 12082 tablet_service.cc:1467] Processing CreateTablet for tablet e97529f5c0244486b7d946ff4526fe59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.255595 12081 tablet_service.cc:1467] Processing CreateTablet for tablet 575f6e5055d84dc38ef2326cec06fa47 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.255955 12082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97529f5c0244486b7d946ff4526fe59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.256172 12081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575f6e5055d84dc38ef2326cec06fa47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.260754 11952 tablet_service.cc:1467] Processing CreateTablet for tablet 575f6e5055d84dc38ef2326cec06fa47 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.260772 12212 tablet_service.cc:1467] Processing CreateTablet for tablet e97529f5c0244486b7d946ff4526fe59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:35.261116 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575f6e5055d84dc38ef2326cec06fa47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.261133 12212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97529f5c0244486b7d946ff4526fe59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:35.267212 12930 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:35.267858 12929 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:35.268918 12930 tablet_bootstrap.cc:654] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.269448 12929 tablet_bootstrap.cc:654] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.270078 12931 tablet_bootstrap.cc:492] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:35.271231 12930 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:35.271354 12930 ts_tablet_manager.cc:1367] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 20:09:35.271458 12929 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:35.271605 12929 ts_tablet_manager.cc:1367] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:09:35.271682 12931 tablet_bootstrap.cc:654] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.271962 12930 raft_consensus.cc:348] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.272131 12930 raft_consensus.cc:374] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.272220 12930 raft_consensus.cc:729] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.272311 12929 raft_consensus.cc:348] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.272547 12929 raft_consensus.cc:374] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.272430 12930 consensus_queue.cc:260] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.272637 12929 raft_consensus.cc:729] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.272974 12930 ts_tablet_manager.cc:1398] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:09:35.272858 12929 consensus_queue.cc:260] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.273382 12929 ts_tablet_manager.cc:1398] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:35.273406 12930 tablet_bootstrap.cc:492] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:35.273651 12929 tablet_bootstrap.cc:492] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:35.274808 12930 tablet_bootstrap.cc:654] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.275065 12929 tablet_bootstrap.cc:654] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.281374 12930 tablet_bootstrap.cc:492] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:35.281507 12930 ts_tablet_manager.cc:1367] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.003s
I20240501 20:09:35.282027 12930 raft_consensus.cc:348] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:35.282177 12930 raft_consensus.cc:374] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.282253 12930 raft_consensus.cc:729] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.282567 12930 consensus_queue.cc:260] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:35.283082 12930 ts_tablet_manager.cc:1398] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240501 20:09:35.283325 12930 tablet_bootstrap.cc:492] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:35.283394 12931 tablet_bootstrap.cc:492] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:35.283532 12931 ts_tablet_manager.cc:1367] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.007s
I20240501 20:09:35.284118 12929 tablet_bootstrap.cc:492] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:35.284322 12929 ts_tablet_manager.cc:1367] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.004s
I20240501 20:09:35.284256 12931 raft_consensus.cc:348] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:35.284447 12931 raft_consensus.cc:374] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.284551 12931 raft_consensus.cc:729] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.284749 12930 tablet_bootstrap.cc:654] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:35.284807 12931 consensus_queue.cc:260] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:35.285431 12931 ts_tablet_manager.cc:1398] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:09:35.286020 12929 raft_consensus.cc:348] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.286775 12929 raft_consensus.cc:374] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.286870 12929 raft_consensus.cc:729] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.287011 12930 tablet_bootstrap.cc:492] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:35.287181 12930 ts_tablet_manager.cc:1367] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:09:35.287128 12929 consensus_queue.cc:260] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.287577 12929 ts_tablet_manager.cc:1398] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 20:09:35.287954 12930 raft_consensus.cc:348] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.288137 12930 raft_consensus.cc:374] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:35.288234 12930 raft_consensus.cc:729] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:35.288509 12930 consensus_queue.cc:260] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.289070 12930 ts_tablet_manager.cc:1398] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:35.307860 12899 raft_consensus.cc:482] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:35.307978 12899 raft_consensus.cc:504] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.308629 12899 leader_election.cc:288] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:35.314193 11973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705" is_pre_election: true
I20240501 20:09:35.314405 11973 raft_consensus.cc:2454] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 0.
I20240501 20:09:35.314749 12036 leader_election.cc:302] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:35.314994 12899 raft_consensus.cc:2781] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:35.315107 12899 raft_consensus.cc:482] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:35.315186 12899 raft_consensus.cc:3037] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.317164 12899 raft_consensus.cc:504] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.317596 12899 leader_election.cc:288] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:35.318024 11973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:35.318193 11973 raft_consensus.cc:3037] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.320003 11973 raft_consensus.cc:2454] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 1.
I20240501 20:09:35.320343 12036 leader_election.cc:302] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:35.320614 12899 raft_consensus.cc:2781] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:35.320747 12899 raft_consensus.cc:686] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:35.321018 12899 consensus_queue.cc:237] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.323227 11830 catalog_manager.cc:5484] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:35.382254 12899 raft_consensus.cc:482] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:35.382444 12899 raft_consensus.cc:504] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:35.382889 12899 leader_election.cc:288] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:35.383338 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97529f5c0244486b7d946ff4526fe59" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad" is_pre_election: true
I20240501 20:09:35.383533 12233 raft_consensus.cc:2454] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 0.
I20240501 20:09:35.383925 12036 leader_election.cc:302] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:35.384191 12899 raft_consensus.cc:2781] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:35.384275 12899 raft_consensus.cc:482] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:35.384374 12899 raft_consensus.cc:3037] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.386155 12899 raft_consensus.cc:504] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:35.386556 12899 leader_election.cc:288] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Requested vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:35.386945 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97529f5c0244486b7d946ff4526fe59" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad"
I20240501 20:09:35.387089 12233 raft_consensus.cc:3037] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.389043 12233 raft_consensus.cc:2454] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 1.
I20240501 20:09:35.389366 12036 leader_election.cc:302] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:35.389597 12899 raft_consensus.cc:2781] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:35.389734 12899 raft_consensus.cc:686] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:35.389940 12899 consensus_queue.cc:237] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:35.391891 11830 catalog_manager.cc:5484] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:35.519501 12899 raft_consensus.cc:482] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:35.519675 12899 raft_consensus.cc:504] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.520224 12899 leader_election.cc:288] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:35.520723 11973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575f6e5055d84dc38ef2326cec06fa47" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705" is_pre_election: true
I20240501 20:09:35.520946 11973 raft_consensus.cc:2454] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 0.
I20240501 20:09:35.521389 12036 leader_election.cc:302] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:35.521663 12899 raft_consensus.cc:2781] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:35.521776 12899 raft_consensus.cc:482] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:35.521849 12899 raft_consensus.cc:3037] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.523734 12899 raft_consensus.cc:504] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.524142 12899 leader_election.cc:288] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:35.524587 11973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575f6e5055d84dc38ef2326cec06fa47" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:35.524742 11973 raft_consensus.cc:3037] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:35.526643 11973 raft_consensus.cc:2454] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 1.
I20240501 20:09:35.526971 12036 leader_election.cc:302] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:35.527246 12899 raft_consensus.cc:2781] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:35.527376 12899 raft_consensus.cc:686] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:35.527642 12899 consensus_queue.cc:237] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:35.529573 11830 catalog_manager.cc:5484] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:35.761893 12899 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:35.850911 12901 consensus_queue.cc:1035] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:35.869622 12942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:35.869874 12942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:35.877038 12942 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:09:35.978085 11973 raft_consensus.cc:1261] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:35.979270 12899 consensus_queue.cc:1035] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:35.993503 12975 mvcc.cc:205] Tried to move back new op lower bound from 7022977744800620544 to 7022977742962536448. Current Snapshot: MvccSnapshot[applied={T|T < 7022977744800620544}]

INSERT report
    rows total: 2000
    time total: 120.304 ms
  time per row: 0.0601519 ms
I20240501 20:09:36.124789 11828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49824:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:36.132478 12212 tablet_service.cc:1467] Processing CreateTablet for tablet 321fb505be2f4263a79c9bcba4be6616 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b40a568bc0e8467881e6ef108d91304f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.132802 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 321fb505be2f4263a79c9bcba4be6616 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b40a568bc0e8467881e6ef108d91304f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.132910 12212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321fb505be2f4263a79c9bcba4be6616. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.132882 12083 tablet_service.cc:1467] Processing CreateTablet for tablet f25dcb94e89c4c42979fdb39523b7fd2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b40a568bc0e8467881e6ef108d91304f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.133208 11953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321fb505be2f4263a79c9bcba4be6616. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.133234 12083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25dcb94e89c4c42979fdb39523b7fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.134651 12213 tablet_service.cc:1467] Processing CreateTablet for tablet f25dcb94e89c4c42979fdb39523b7fd2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b40a568bc0e8467881e6ef108d91304f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.134786 11952 tablet_service.cc:1467] Processing CreateTablet for tablet 0faf9b33582449ee9ffa353ae6ea3d6a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b40a568bc0e8467881e6ef108d91304f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.134996 12213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25dcb94e89c4c42979fdb39523b7fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.135092 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faf9b33582449ee9ffa353ae6ea3d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.138522 12211 tablet_service.cc:1467] Processing CreateTablet for tablet 0faf9b33582449ee9ffa353ae6ea3d6a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b40a568bc0e8467881e6ef108d91304f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.138844 12211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faf9b33582449ee9ffa353ae6ea3d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.144752 12995 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:36.146399 12995 tablet_bootstrap.cc:654] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.149802 12994 tablet_bootstrap.cc:492] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:36.150429 12993 tablet_bootstrap.cc:492] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:36.151671 12995 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:36.151834 12995 ts_tablet_manager.cc:1367] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.003s
I20240501 20:09:36.151856 12993 tablet_bootstrap.cc:654] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.151854 12994 tablet_bootstrap.cc:654] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.152524 12995 raft_consensus.cc:348] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.152680 12995 raft_consensus.cc:374] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.152812 12995 raft_consensus.cc:729] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.153103 12995 consensus_queue.cc:260] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.153607 12993 tablet_bootstrap.cc:492] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:36.153736 12993 ts_tablet_manager.cc:1367] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.003s
I20240501 20:09:36.153733 12995 ts_tablet_manager.cc:1398] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:36.153988 12995 tablet_bootstrap.cc:492] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:36.154261 12994 tablet_bootstrap.cc:492] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:36.154439 12994 ts_tablet_manager.cc:1367] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240501 20:09:36.154704 12993 raft_consensus.cc:348] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:36.154903 12993 raft_consensus.cc:374] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.155009 12993 raft_consensus.cc:729] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.155158 12994 raft_consensus.cc:348] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.155256 12993 consensus_queue.cc:260] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:36.155355 12994 raft_consensus.cc:374] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.155391 12995 tablet_bootstrap.cc:654] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.155436 12994 raft_consensus.cc:729] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.155674 12994 consensus_queue.cc:260] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.155887 12993 ts_tablet_manager.cc:1398] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:36.156272 12994 ts_tablet_manager.cc:1398] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:09:36.156388 12993 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:36.157765 12993 tablet_bootstrap.cc:654] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.160665 12995 tablet_bootstrap.cc:492] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:36.160797 12995 ts_tablet_manager.cc:1367] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.002s
I20240501 20:09:36.160934 12993 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:36.161065 12993 ts_tablet_manager.cc:1367] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.003s
I20240501 20:09:36.161329 12995 raft_consensus.cc:348] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:36.161527 12995 raft_consensus.cc:374] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.161598 12995 raft_consensus.cc:729] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.161657 12993 raft_consensus.cc:348] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.161787 12993 raft_consensus.cc:374] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.161829 12993 raft_consensus.cc:729] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.161782 12995 consensus_queue.cc:260] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:36.161993 12993 consensus_queue.cc:260] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.162201 12995 ts_tablet_manager.cc:1398] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.001s
I20240501 20:09:36.163297 12993 ts_tablet_manager.cc:1398] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:36.163540 12993 tablet_bootstrap.cc:492] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:36.164914 12993 tablet_bootstrap.cc:654] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.166637 12993 tablet_bootstrap.cc:492] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:36.166752 12993 ts_tablet_manager.cc:1367] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.001s
I20240501 20:09:36.167482 12993 raft_consensus.cc:348] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.167652 12993 raft_consensus.cc:374] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.167734 12993 raft_consensus.cc:729] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.167939 12993 consensus_queue.cc:260] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.168380 12993 ts_tablet_manager.cc:1398] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:09:36.247195 12983 raft_consensus.cc:482] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.247341 12983 raft_consensus.cc:504] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.247871 12983 leader_election.cc:288] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:36.248347 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25dcb94e89c4c42979fdb39523b7fd2" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad" is_pre_election: true
I20240501 20:09:36.248554 12233 raft_consensus.cc:2454] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 0.
I20240501 20:09:36.248932 12036 leader_election.cc:302] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:36.249186 12983 raft_consensus.cc:2781] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.249279 12983 raft_consensus.cc:482] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.249356 12983 raft_consensus.cc:3037] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.251194 12983 raft_consensus.cc:504] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.251583 12983 leader_election.cc:288] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Requested vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:36.252027 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25dcb94e89c4c42979fdb39523b7fd2" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad"
I20240501 20:09:36.252189 12233 raft_consensus.cc:3037] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.253891 12233 raft_consensus.cc:2454] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 1.
I20240501 20:09:36.254222 12036 leader_election.cc:302] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:36.254493 12983 raft_consensus.cc:2781] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.254621 12983 raft_consensus.cc:686] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:36.254849 12983 consensus_queue.cc:237] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.257064 11830 catalog_manager.cc:5484] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:36.263674 12858 raft_consensus.cc:482] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.263793 12858 raft_consensus.cc:504] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:36.264181 12858 leader_election.cc:288] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:36.264679 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321fb505be2f4263a79c9bcba4be6616" candidate_uuid: "18717c9754d04918815985cf07c1ddad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705" is_pre_election: true
I20240501 20:09:36.264878 11972 raft_consensus.cc:2454] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18717c9754d04918815985cf07c1ddad in term 0.
I20240501 20:09:36.265256 12166 leader_election.cc:302] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:36.265496 12858 raft_consensus.cc:2781] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.265584 12858 raft_consensus.cc:482] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.265643 12858 raft_consensus.cc:3037] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.267393 12858 raft_consensus.cc:504] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:36.267808 12858 leader_election.cc:288] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:36.268191 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321fb505be2f4263a79c9bcba4be6616" candidate_uuid: "18717c9754d04918815985cf07c1ddad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:36.268342 11972 raft_consensus.cc:3037] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.270098 11972 raft_consensus.cc:2454] T 321fb505be2f4263a79c9bcba4be6616 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18717c9754d04918815985cf07c1ddad in term 1.
I20240501 20:09:36.270427 12166 leader_election.cc:302] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:36.270694 12858 raft_consensus.cc:2781] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.271085 12858 raft_consensus.cc:686] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Becoming Leader. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Running, Role: LEADER
I20240501 20:09:36.271356 12858 consensus_queue.cc:237] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:36.273569 11830 catalog_manager.cc:5484] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad reported cstate change: term changed from 0 to 1, leader changed from <none> to 18717c9754d04918815985cf07c1ddad (127.8.68.195). New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:36.446487 13004 raft_consensus.cc:482] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.446686 13004 raft_consensus.cc:504] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.447288 13004 leader_election.cc:288] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:36.447695 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faf9b33582449ee9ffa353ae6ea3d6a" candidate_uuid: "18717c9754d04918815985cf07c1ddad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705" is_pre_election: true
I20240501 20:09:36.447935 11972 raft_consensus.cc:2454] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18717c9754d04918815985cf07c1ddad in term 0.
I20240501 20:09:36.448268 12166 leader_election.cc:302] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:36.448503 13004 raft_consensus.cc:2781] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.448632 13004 raft_consensus.cc:482] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.448715 13004 raft_consensus.cc:3037] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.451587 13004 raft_consensus.cc:504] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.452150 13004 leader_election.cc:288] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:36.452536 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faf9b33582449ee9ffa353ae6ea3d6a" candidate_uuid: "18717c9754d04918815985cf07c1ddad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:36.452759 11972 raft_consensus.cc:3037] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.455569 11972 raft_consensus.cc:2454] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18717c9754d04918815985cf07c1ddad in term 1.
I20240501 20:09:36.455927 12166 leader_election.cc:302] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:36.456169 13004 raft_consensus.cc:2781] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.456295 13004 raft_consensus.cc:686] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Becoming Leader. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Running, Role: LEADER
I20240501 20:09:36.456538 13004 consensus_queue.cc:237] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:36.459199 11830 catalog_manager.cc:5484] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad reported cstate change: term changed from 0 to 1, leader changed from <none> to 18717c9754d04918815985cf07c1ddad (127.8.68.195). New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:36.768013 12983 consensus_queue.cc:1035] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:36.815554 13007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:36.815809 13007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:36.822944 13007 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:36.831729 13004 consensus_queue.cc:1035] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 20:09:36.916671 13004 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 160.469 ms
  time per row: 0.0802347 ms
I20240501 20:09:37.113754 11832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49824:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:37.121259 11952 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb828d8511f40c78004d0aaba843df5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.121697 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb828d8511f40c78004d0aaba843df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.121719 12081 tablet_service.cc:1467] Processing CreateTablet for tablet 38b8e2865d134700ad0959372afc6670 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.121950 12083 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb828d8511f40c78004d0aaba843df5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.122084 12081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b8e2865d134700ad0959372afc6670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.122347 12083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb828d8511f40c78004d0aaba843df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.122916 12082 tablet_service.cc:1467] Processing CreateTablet for tablet 67043e58e5fd4233bc945432eeae1227 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.123251 12082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67043e58e5fd4233bc945432eeae1227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.126348 12212 tablet_service.cc:1467] Processing CreateTablet for tablet 38b8e2865d134700ad0959372afc6670 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.126741 12212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b8e2865d134700ad0959372afc6670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.127415 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 67043e58e5fd4233bc945432eeae1227 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.127785 11953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67043e58e5fd4233bc945432eeae1227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.134752 13062 tablet_bootstrap.cc:492] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:37.135598 13061 tablet_bootstrap.cc:492] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:37.135862 13060 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:37.136520 13062 tablet_bootstrap.cc:654] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.137423 13060 tablet_bootstrap.cc:654] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.137648 13061 tablet_bootstrap.cc:654] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.139250 13062 tablet_bootstrap.cc:492] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:37.139386 13062 ts_tablet_manager.cc:1367] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240501 20:09:37.139591 13060 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:37.139772 13060 ts_tablet_manager.cc:1367] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:09:37.140096 13061 tablet_bootstrap.cc:492] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:37.140084 13062 raft_consensus.cc:348] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.140215 13061 ts_tablet_manager.cc:1367] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 20:09:37.140272 13062 raft_consensus.cc:374] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.140370 13062 raft_consensus.cc:729] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.140435 13060 raft_consensus.cc:348] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.140610 13060 raft_consensus.cc:374] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.140591 13062 consensus_queue.cc:260] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.140697 13060 raft_consensus.cc:729] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.140861 13061 raft_consensus.cc:348] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.140882 13060 consensus_queue.cc:260] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.141024 13061 raft_consensus.cc:374] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.141093 13062 ts_tablet_manager.cc:1398] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:09:37.141119 13061 raft_consensus.cc:729] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.141333 13062 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:37.141330 13061 consensus_queue.cc:260] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.141528 13060 ts_tablet_manager.cc:1398] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:37.141839 13061 ts_tablet_manager.cc:1398] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:37.142683 13062 tablet_bootstrap.cc:654] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.141767 13060 tablet_bootstrap.cc:492] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:37.144975 13060 tablet_bootstrap.cc:654] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.146972 13062 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:37.147122 13062 ts_tablet_manager.cc:1367] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 20:09:37.147819 13062 raft_consensus.cc:348] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.148000 13062 raft_consensus.cc:374] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.148072 13062 raft_consensus.cc:729] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.148265 13062 consensus_queue.cc:260] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.148747 13062 ts_tablet_manager.cc:1398] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:37.148823 13060 tablet_bootstrap.cc:492] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:37.148957 13060 ts_tablet_manager.cc:1367] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240501 20:09:37.149441 13060 raft_consensus.cc:348] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.149593 13060 raft_consensus.cc:374] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.149659 13060 raft_consensus.cc:729] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.149857 13060 consensus_queue.cc:260] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.150225 13060 ts_tablet_manager.cc:1398] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:09:37.150472 13060 tablet_bootstrap.cc:492] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:37.151746 13060 tablet_bootstrap.cc:654] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.153599 13060 tablet_bootstrap.cc:492] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:37.153724 13060 ts_tablet_manager.cc:1367] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:37.154311 13060 raft_consensus.cc:348] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.154444 13060 raft_consensus.cc:374] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.154527 13060 raft_consensus.cc:729] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.154706 13060 consensus_queue.cc:260] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.155047 13060 ts_tablet_manager.cc:1398] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240501 20:09:37.160073 13009 raft_consensus.cc:482] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.160189 13009 raft_consensus.cc:504] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.160616 13009 leader_election.cc:288] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:37.161100 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67043e58e5fd4233bc945432eeae1227" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705" is_pre_election: true
I20240501 20:09:37.161291 11972 raft_consensus.cc:2454] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 0.
I20240501 20:09:37.161653 12036 leader_election.cc:302] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:37.161895 13009 raft_consensus.cc:2781] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.162026 13009 raft_consensus.cc:482] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.162102 13009 raft_consensus.cc:3037] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.163913 13009 raft_consensus.cc:504] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.164309 13009 leader_election.cc:288] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:37.164775 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67043e58e5fd4233bc945432eeae1227" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:37.164944 11972 raft_consensus.cc:3037] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.166621 11972 raft_consensus.cc:2454] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 1.
I20240501 20:09:37.166992 12036 leader_election.cc:302] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:37.167260 13009 raft_consensus.cc:2781] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.167394 13009 raft_consensus.cc:686] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:37.167665 13009 consensus_queue.cc:237] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.169888 11832 catalog_manager.cc:5484] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:37.188324 12983 raft_consensus.cc:482] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.188447 12983 raft_consensus.cc:504] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.188865 12983 leader_election.cc:288] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:37.189302 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb828d8511f40c78004d0aaba843df5" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705" is_pre_election: true
I20240501 20:09:37.189503 11972 raft_consensus.cc:2454] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 0.
I20240501 20:09:37.189831 12036 leader_election.cc:302] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:37.190052 12983 raft_consensus.cc:2781] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.190177 12983 raft_consensus.cc:482] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.190304 12983 raft_consensus.cc:3037] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.192948 12983 raft_consensus.cc:504] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.193428 12983 leader_election.cc:288] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:37.193825 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb828d8511f40c78004d0aaba843df5" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:37.194001 11972 raft_consensus.cc:3037] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.196480 11972 raft_consensus.cc:2454] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 1.
I20240501 20:09:37.196835 12036 leader_election.cc:302] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:37.197122 12983 raft_consensus.cc:2781] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.197273 12983 raft_consensus.cc:686] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:37.197566 12983 consensus_queue.cc:237] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.200168 11832 catalog_manager.cc:5484] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:37.285735 12983 raft_consensus.cc:482] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.285907 12983 raft_consensus.cc:504] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.286406 12983 leader_election.cc:288] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:37.286866 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b8e2865d134700ad0959372afc6670" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad" is_pre_election: true
I20240501 20:09:37.287066 12233 raft_consensus.cc:2454] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 0.
I20240501 20:09:37.287462 12036 leader_election.cc:302] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:37.287742 12983 raft_consensus.cc:2781] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.287820 12983 raft_consensus.cc:482] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.287912 12983 raft_consensus.cc:3037] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.289803 12983 raft_consensus.cc:504] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.290244 12983 leader_election.cc:288] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Requested vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:37.290668 12233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b8e2865d134700ad0959372afc6670" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad"
I20240501 20:09:37.290812 12233 raft_consensus.cc:3037] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.292682 12233 raft_consensus.cc:2454] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 1.
I20240501 20:09:37.293009 12036 leader_election.cc:302] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:37.293273 12983 raft_consensus.cc:2781] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.293448 12983 raft_consensus.cc:686] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:37.293704 12983 consensus_queue.cc:237] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.296104 11832 catalog_manager.cc:5484] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:37.627382 13072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:37.627643 13072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:37.634537 13072 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 20:09:37.679565 12983 consensus_queue.cc:1035] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.699923 12983 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.788717 12233 raft_consensus.cc:1261] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:37.789450 13101 consensus_queue.cc:1035] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 192.765 ms
  time per row: 0.0963826 ms
I20240501 20:09:37.957655 11829 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49824:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:09:37.965942 12212 tablet_service.cc:1467] Processing CreateTablet for tablet f8f9c24067b442299db6fa5831feb56d (DEFAULT_TABLE table=rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.966413 12212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f9c24067b442299db6fa5831feb56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.966259 11953 tablet_service.cc:1467] Processing CreateTablet for tablet f8f9c24067b442299db6fa5831feb56d (DEFAULT_TABLE table=rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.966670 12083 tablet_service.cc:1467] Processing CreateTablet for tablet 8baf6913f38a42c3a369cd673ff9438d (DEFAULT_TABLE table=rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.966825 11953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f9c24067b442299db6fa5831feb56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.967023 12083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baf6913f38a42c3a369cd673ff9438d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.967167 11952 tablet_service.cc:1467] Processing CreateTablet for tablet d847215537454b4ba9e8a8b87ec91699 (DEFAULT_TABLE table=rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.967502 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d847215537454b4ba9e8a8b87ec91699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.974509 12213 tablet_service.cc:1467] Processing CreateTablet for tablet 8baf6913f38a42c3a369cd673ff9438d (DEFAULT_TABLE table=rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.974902 12213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baf6913f38a42c3a369cd673ff9438d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.978348 13124 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:37.978632 12211 tablet_service.cc:1467] Processing CreateTablet for tablet d847215537454b4ba9e8a8b87ec91699 (DEFAULT_TABLE table=rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.978987 12211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d847215537454b4ba9e8a8b87ec91699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.981779 13124 tablet_bootstrap.cc:654] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.983855 13123 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: Bootstrap starting.
I20240501 20:09:37.986343 13123 tablet_bootstrap.cc:654] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.987048 13124 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:37.987175 13124 ts_tablet_manager.cc:1367] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240501 20:09:37.987706 13124 raft_consensus.cc:348] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:37.988031 13124 raft_consensus.cc:374] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.988140 13124 raft_consensus.cc:729] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.988251 13122 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:37.988541 13124 consensus_queue.cc:260] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:37.989092 13124 ts_tablet_manager.cc:1398] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:37.989663 13122 tablet_bootstrap.cc:654] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.989924 13124 tablet_bootstrap.cc:492] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705: Bootstrap starting.
I20240501 20:09:37.991127 13124 tablet_bootstrap.cc:654] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.991395 13122 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:37.991513 13122 ts_tablet_manager.cc:1367] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:37.992165 13122 raft_consensus.cc:348] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:37.992329 13122 raft_consensus.cc:374] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.992412 13122 raft_consensus.cc:729] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.992611 13122 consensus_queue.cc:260] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:37.992756 13124 tablet_bootstrap.cc:492] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705: No bootstrap required, opened a new log
I20240501 20:09:37.992877 13124 ts_tablet_manager.cc:1367] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:37.993448 13124 raft_consensus.cc:348] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.993606 13122 ts_tablet_manager.cc:1398] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:37.993634 13124 raft_consensus.cc:374] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.993707 13124 raft_consensus.cc:729] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.993845 13122 tablet_bootstrap.cc:492] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:37.993913 13124 consensus_queue.cc:260] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.994431 13124 ts_tablet_manager.cc:1398] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240501 20:09:37.995841 13122 tablet_bootstrap.cc:654] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.997736 13123 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: No bootstrap required, opened a new log
I20240501 20:09:37.997874 13123 ts_tablet_manager.cc:1367] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.000s
I20240501 20:09:37.998049 13122 tablet_bootstrap.cc:492] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:37.998186 13122 ts_tablet_manager.cc:1367] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:09:37.998600 13123 raft_consensus.cc:348] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.998795 13123 raft_consensus.cc:374] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.998895 13123 raft_consensus.cc:729] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.998862 13122 raft_consensus.cc:348] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:37.999181 13122 raft_consensus.cc:374] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.999104 13123 consensus_queue.cc:260] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:37.999259 13122 raft_consensus.cc:729] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.999594 13123 ts_tablet_manager.cc:1398] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:38.000010 13122 consensus_queue.cc:260] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:38.000418 13122 ts_tablet_manager.cc:1398] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:09:38.000617 13122 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad: Bootstrap starting.
I20240501 20:09:38.001711 13122 tablet_bootstrap.cc:654] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:38.003391 13122 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad: No bootstrap required, opened a new log
I20240501 20:09:38.003499 13122 ts_tablet_manager.cc:1367] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 20:09:38.003970 13122 raft_consensus.cc:348] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:38.004092 13122 raft_consensus.cc:374] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:38.004173 13122 raft_consensus.cc:729] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Initialized, Role: FOLLOWER
I20240501 20:09:38.004341 13122 consensus_queue.cc:260] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:38.004738 13122 ts_tablet_manager.cc:1398] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:09:38.363025 13050 raft_consensus.cc:482] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:38.363185 13050 raft_consensus.cc:504] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:38.363713 13050 leader_election.cc:288] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:38.364187 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d847215537454b4ba9e8a8b87ec91699" candidate_uuid: "18717c9754d04918815985cf07c1ddad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705" is_pre_election: true
I20240501 20:09:38.364372 11972 raft_consensus.cc:2454] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18717c9754d04918815985cf07c1ddad in term 0.
I20240501 20:09:38.364473 12883 raft_consensus.cc:482] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:38.364630 12883 raft_consensus.cc:504] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:38.364797 12166 leader_election.cc:302] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:38.365064 13050 raft_consensus.cc:2781] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:38.365146 12883 leader_election.cc:288] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:38.365182 13050 raft_consensus.cc:482] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:38.365255 13050 raft_consensus.cc:3037] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.365654 12231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f9c24067b442299db6fa5831feb56d" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad" is_pre_election: true
I20240501 20:09:38.365831 12231 raft_consensus.cc:2454] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 0.
I20240501 20:09:38.366210 11906 leader_election.cc:302] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:38.366461 12883 raft_consensus.cc:2781] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:38.366595 12883 raft_consensus.cc:482] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:38.366688 12883 raft_consensus.cc:3037] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.367739 13050 raft_consensus.cc:504] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:38.368201 13050 leader_election.cc:288] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:38.368594 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d847215537454b4ba9e8a8b87ec91699" candidate_uuid: "18717c9754d04918815985cf07c1ddad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:38.368746 11972 raft_consensus.cc:3037] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.369151 12883 raft_consensus.cc:504] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:38.369560 12883 leader_election.cc:288] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 election: Requested vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:38.370002 12231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f9c24067b442299db6fa5831feb56d" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad"
I20240501 20:09:38.370147 12231 raft_consensus.cc:3037] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.370736 11972 raft_consensus.cc:2454] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18717c9754d04918815985cf07c1ddad in term 1.
I20240501 20:09:38.371068 12166 leader_election.cc:302] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:38.371335 13050 raft_consensus.cc:2781] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:38.371719 13050 raft_consensus.cc:686] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Becoming Leader. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Running, Role: LEADER
I20240501 20:09:38.372001 13050 consensus_queue.cc:237] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } }
I20240501 20:09:38.372525 12231 raft_consensus.cc:2454] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 1.
I20240501 20:09:38.372879 11906 leader_election.cc:302] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:38.373152 12883 raft_consensus.cc:2781] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:38.373507 12883 raft_consensus.cc:686] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Becoming Leader. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Running, Role: LEADER
I20240501 20:09:38.373754 12883 consensus_queue.cc:237] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } }
I20240501 20:09:38.375447 11829 catalog_manager.cc:5484] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad reported cstate change: term changed from 0 to 1, leader changed from <none> to 18717c9754d04918815985cf07c1ddad (127.8.68.195). New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:38.376215 11828 catalog_manager.cc:5484] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc71b180f72b4c4697d28b13b363d705 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:38.426344 12983 raft_consensus.cc:482] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:38.426478 12983 raft_consensus.cc:504] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:38.426901 12983 leader_election.cc:288] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:38.427289 12231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf6913f38a42c3a369cd673ff9438d" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad" is_pre_election: true
I20240501 20:09:38.427453 12231 raft_consensus.cc:2454] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 0.
I20240501 20:09:38.427755 12036 leader_election.cc:302] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:38.428001 12983 raft_consensus.cc:2781] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:38.428120 12983 raft_consensus.cc:482] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:38.428207 12983 raft_consensus.cc:3037] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.429934 12983 raft_consensus.cc:504] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:38.430327 12983 leader_election.cc:288] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Requested vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:38.430667 12231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf6913f38a42c3a369cd673ff9438d" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18717c9754d04918815985cf07c1ddad"
I20240501 20:09:38.430837 12231 raft_consensus.cc:3037] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.432473 12231 raft_consensus.cc:2454] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 1.
I20240501 20:09:38.432791 12036 leader_election.cc:302] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 34f8af7f2d4245899a7d8fd5012a3721; no voters: 
I20240501 20:09:38.433045 12983 raft_consensus.cc:2781] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:38.433347 12983 raft_consensus.cc:686] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:38.433560 12983 consensus_queue.cc:237] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } }
I20240501 20:09:38.435566 11829 catalog_manager.cc:5484] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } }
W20240501 20:09:38.837731 13138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:38.837991 13138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:38.844897 13138 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 20:09:38.914351 12883 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.980610 13050 consensus_queue.cc:1035] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:39.000872 12231 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:39.001431 12983 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 212.246 ms
  time per row: 0.106123 ms
I20240501 20:09:39.186815  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:36351
--local_ip_for_outbound_sockets=127.8.68.196
--tserver_master_addrs=127.8.68.254:46165
--webserver_port=41693
--webserver_interface=127.8.68.196
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:09:39.512398 13195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:39.512656 13195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:39.512763 13195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:39.520134 13195 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:39.520241 13195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:39.520350 13195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:09:39.529052 13195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:36351
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=41693
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:39.530123 13195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:39.531749 13195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:39.543215 13200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.543772 13201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.544173 13203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:39.544700 13195 server_base.cc:1034] running on GCE node
I20240501 20:09:39.545192 13195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:39.546180 13195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:39.547394 13195 hybrid_clock.cc:648] HybridClock initialized: now 1714594179547335 us; error 48 us; skew 500 ppm
I20240501 20:09:39.547734 13195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:39.550040 13195 webserver.cc:460] Webserver started at http://127.8.68.196:41693/ using document root <none> and password file <none>
I20240501 20:09:39.550715 13195 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:39.550824 13195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:39.555466 13195 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.002s
I20240501 20:09:39.557847 13209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:39.559072 13195 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:09:39.559211 13195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3b95f8c6589e473bb21db7681e227638"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:39.559942 13195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:39.588876 13195 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:39.589659 13195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:39.589905 13195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:39.590778 13195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:39.592201 13195 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:39.592275 13195 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:39.592387 13195 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:39.592447 13195 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:39.625667 13195 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:36351
I20240501 20:09:39.625716 13321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:36351 every 8 connection(s)
I20240501 20:09:39.627306 13195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:09:39.628696  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 13195
I20240501 20:09:39.629392  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:43403
--local_ip_for_outbound_sockets=127.8.68.197
--tserver_master_addrs=127.8.68.254:46165
--webserver_port=39737
--webserver_interface=127.8.68.197
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:39.643580 13322 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:39.643841 13322 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:39.644531 13322 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:39.646203 11827 ts_manager.cc:194] Re-registered known tserver with Master: 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351)
I20240501 20:09:39.647395 11827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:54889
W20240501 20:09:39.942587 13326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:39.942848 13326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:39.942950 13326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:39.949877 13326 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:39.949977 13326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:39.950201 13326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:09:39.958907 13326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:43403
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=39737
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:39.960070 13326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:39.961634 13326 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:39.972641 13331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.973121 13332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.973676 13334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:39.974151 13326 server_base.cc:1034] running on GCE node
I20240501 20:09:39.974717 13326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:39.975577 13326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:39.976753 13326 hybrid_clock.cc:648] HybridClock initialized: now 1714594179976712 us; error 42 us; skew 500 ppm
I20240501 20:09:39.977036 13326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:39.979044 13326 webserver.cc:460] Webserver started at http://127.8.68.197:39737/ using document root <none> and password file <none>
I20240501 20:09:39.979696 13326 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:39.979813 13326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:39.984571 13326 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240501 20:09:39.987246 13340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:39.988482 13326 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:09:39.988623 13326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "535367472879403fa999dd00a155f042"
format_stamp: "Formatted at 2024-05-01 20:09:28 on dist-test-slave-08p9"
I20240501 20:09:39.989428 13326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:40.008481 13326 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:40.009176 13326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:40.009433 13326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:40.010250 13326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:40.011742 13326 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:40.011832 13326 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:40.011951 13326 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:40.012014 13326 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:40.039306 13326 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:43403
I20240501 20:09:40.039345 13452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:43403 every 8 connection(s)
I20240501 20:09:40.040606 13326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:09:40.050855  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 13326
I20240501 20:09:40.051440  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:35531
--local_ip_for_outbound_sockets=127.8.68.198
--tserver_master_addrs=127.8.68.254:46165
--webserver_port=39883
--webserver_interface=127.8.68.198
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:40.052213 13453 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:40.052464 13453 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:40.053159 13453 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:40.054404 11827 ts_manager.cc:194] Re-registered known tserver with Master: 535367472879403fa999dd00a155f042 (127.8.68.197:43403)
I20240501 20:09:40.055141 11827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:51939
W20240501 20:09:40.375664 13456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:40.375910 13456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:40.376019 13456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:40.383126 13456 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:40.383231 13456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:40.383373 13456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240501 20:09:40.392104 13456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:35531
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=39883
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:40.393229 13456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:40.394873 13456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:40.405910 13461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:40.406325 13462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:40.406831 13464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:40.407341 13456 server_base.cc:1034] running on GCE node
I20240501 20:09:40.407805 13456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:40.408592 13456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:40.409770 13456 hybrid_clock.cc:648] HybridClock initialized: now 1714594180409724 us; error 38 us; skew 500 ppm
I20240501 20:09:40.410035 13456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:40.412006 13456 webserver.cc:460] Webserver started at http://127.8.68.198:39883/ using document root <none> and password file <none>
I20240501 20:09:40.412642 13456 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:40.412757 13456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:40.417390 13456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.001s
I20240501 20:09:40.419798 13470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:40.420909 13456 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:40.421039 13456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "463bd46a1f6e4e89a3040c4cf50dd919"
format_stamp: "Formatted at 2024-05-01 20:09:29 on dist-test-slave-08p9"
I20240501 20:09:40.421775 13456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:40.440179 13456 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:40.440802 13456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:40.441052 13456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:40.441833 13456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:40.443161 13456 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:40.443239 13456 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:40.443356 13456 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:40.443408 13456 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:40.470887 13456 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:35531
I20240501 20:09:40.470924 13583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:35531 every 8 connection(s)
I20240501 20:09:40.472162 13456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:09:40.481475  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 13456
I20240501 20:09:40.482133  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:40105
--local_ip_for_outbound_sockets=127.8.68.199
--tserver_master_addrs=127.8.68.254:46165
--webserver_port=42839
--webserver_interface=127.8.68.199
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:40.483791 13584 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:40.484076 13584 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:40.484930 13584 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:40.486526 11827 ts_manager.cc:194] Re-registered known tserver with Master: 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198:35531)
I20240501 20:09:40.487596 11827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:60845
I20240501 20:09:40.649724 13322 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
W20240501 20:09:40.792824 13587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:40.793076 13587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:40.793211 13587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:40.800204 13587 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:40.800314 13587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:40.800448 13587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240501 20:09:40.808856 13587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:40105
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=42839
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:40.809942 13587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:40.811506 13587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:40.822592 13592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:40.823012 13595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:40.823366 13593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:40.823992 13587 server_base.cc:1034] running on GCE node
I20240501 20:09:40.824549 13587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:40.825603 13587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:40.826786 13587 hybrid_clock.cc:648] HybridClock initialized: now 1714594180826738 us; error 43 us; skew 500 ppm
I20240501 20:09:40.827067 13587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:40.828990 13587 webserver.cc:460] Webserver started at http://127.8.68.199:42839/ using document root <none> and password file <none>
I20240501 20:09:40.829632 13587 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:40.829746 13587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:40.834386 13587 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.001s
I20240501 20:09:40.836933 13601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:40.838093 13587 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:09:40.838234 13587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "aed522b44a8e420daf65718450541c3c"
format_stamp: "Formatted at 2024-05-01 20:09:29 on dist-test-slave-08p9"
I20240501 20:09:40.839018 13587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:40.857894 13587 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:09:40.858583 13587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:40.858827 13587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:40.859614 13587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:40.860939 13587 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:40.861022 13587 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:40.861166 13587 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:40.861228 13587 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:09:40.890372 13587 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:40105
I20240501 20:09:40.890463 13713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:40105 every 8 connection(s)
I20240501 20:09:40.891660 13587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:09:40.892858  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 13587
I20240501 20:09:40.893195  8467 test_util.cc:268] Using random seed: 286656984
I20240501 20:09:40.905530 13714 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:09:40.905808 13714 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:40.906798 13714 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:09:40.908195 11827 ts_manager.cc:194] Re-registered known tserver with Master: aed522b44a8e420daf65718450541c3c (127.8.68.199:40105)
I20240501 20:09:40.909106 11827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:38673
I20240501 20:09:40.910229  8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 20:09:40.910732  8467 test_util.cc:268] Using random seed: 286674522
I20240501 20:09:40.924333  8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 20:09:40.924686  8467 test_util.cc:268] Using random seed: 286688476
I20240501 20:09:40.945569  8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 20:09:40.946292  8467 test_util.cc:268] Using random seed: 286710061
I20240501 20:09:40.989495  8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 20:09:40.989964  8467 test_util.cc:268] Using random seed: 286753749
I20240501 20:09:41.057185 13453 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
I20240501 20:09:41.104624  8467 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240501 20:09:41.490447 13584 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
I20240501 20:09:41.911792 13714 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
I20240501 20:09:42.243960 12212 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:42.258503 12083 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:42.320288 13257 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:42.356796 13519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:42.442544 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:42.489815 11952 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:42.525197 13388 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:43.067654 12212 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:43.112352 12083 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:43.112689 13257 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:43.211591 13519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:43.223768 13388 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:43.290778 11952 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:43.307739 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:43.555660 12103 consensus_queue.cc:237] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:43.559466 12229 raft_consensus.cc:1261] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240501 20:09:43.561661 13816 consensus_queue.cc:1035] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240501 20:09:43.566298 13818 raft_consensus.cc:2932] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } } }
I20240501 20:09:43.572338 12233 raft_consensus.cc:2932] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } } }
I20240501 20:09:43.572929 11827 catalog_manager.cc:5484] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index -1 to 59, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:43.594575 12039 consensus_peers.cc:485] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198:35531): Couldn't send request to peer 463bd46a1f6e4e89a3040c4cf50dd919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38b8e2865d134700ad0959372afc6670. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.650444 12103 consensus_queue.cc:237] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.56, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:43.654582 12233 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240501 20:09:43.656282 13782 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240501 20:09:43.659253 13754 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.57: config changed from index -1 to 57, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:43.660260 12232 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index -1 to 57, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
W20240501 20:09:43.666460 12039 consensus_peers.cc:485] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351): Couldn't send request to peer 3b95f8c6589e473bb21db7681e227638. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8baf6913f38a42c3a369cd673ff9438d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.666316 11827 catalog_manager.cc:5484] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index -1 to 57, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:43.716629 12103 consensus_queue.cc:237] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:43.724129 11973 raft_consensus.cc:1261] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240501 20:09:43.725071 13816 consensus_queue.cc:1035] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
W20240501 20:09:43.726447 12036 consensus_peers.cc:485] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer aed522b44a8e420daf65718450541c3c (127.8.68.199:40105): Couldn't send request to peer aed522b44a8e420daf65718450541c3c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 575f6e5055d84dc38ef2326cec06fa47. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.729897 13816 raft_consensus.cc:2932] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } } }
I20240501 20:09:43.731329 11973 raft_consensus.cc:2932] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } } }
I20240501 20:09:43.736485 11825 catalog_manager.cc:5484] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index -1 to 51, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:43.836488 12229 consensus_queue.cc:237] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:43.849684 11973 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240501 20:09:43.852785 13815 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 20:09:43.861398 13815 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index -1 to 81, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } } }
I20240501 20:09:43.867878 11827 catalog_manager.cc:5484] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index -1 to 81, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:43.865335 11970 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } } }
W20240501 20:09:43.888736 12166 consensus_peers.cc:485] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad -> Peer 535367472879403fa999dd00a155f042 (127.8.68.197:43403): Couldn't send request to peer 535367472879403fa999dd00a155f042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0faf9b33582449ee9ffa353ae6ea3d6a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.943045 12232 consensus_queue.cc:237] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
W20240501 20:09:43.960009 12169 consensus_peers.cc:485] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad -> Peer 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198:35531): Couldn't send request to peer 463bd46a1f6e4e89a3040c4cf50dd919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d847215537454b4ba9e8a8b87ec91699. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.963411 11970 raft_consensus.cc:1261] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240501 20:09:43.965251 13815 consensus_queue.cc:1035] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 20:09:43.967469 13802 raft_consensus.cc:2932] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } } }
I20240501 20:09:43.972000 11969 raft_consensus.cc:2932] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } } }
I20240501 20:09:43.974857 11827 catalog_manager.cc:5484] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index -1 to 74, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:44.042333 12103 consensus_queue.cc:237] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
W20240501 20:09:44.044682 12036 consensus_peers.cc:485] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer aed522b44a8e420daf65718450541c3c (127.8.68.199:40105): Couldn't send request to peer aed522b44a8e420daf65718450541c3c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f25dcb94e89c4c42979fdb39523b7fd2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:44.045444 12229 raft_consensus.cc:1261] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 20:09:44.047714 13754 consensus_queue.cc:1035] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240501 20:09:44.051010 13816 raft_consensus.cc:2932] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index -1 to 72, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } } }
I20240501 20:09:44.056545 11825 catalog_manager.cc:5484] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index -1 to 72, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:44.057916 12232 raft_consensus.cc:2932] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index -1 to 72, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } } }
I20240501 20:09:44.064128 13960 ts_tablet_manager.cc:902] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:44.112376 13960 tablet_copy_client.cc:308] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:09:44.152216 13971 ts_tablet_manager.cc:902] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:44.152820 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet 8baf6913f38a42c3a369cd673ff9438d from peer 3b95f8c6589e473bb21db7681e227638 ({username='slave'} at 127.8.68.196:55885)
I20240501 20:09:44.153090 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet 8baf6913f38a42c3a369cd673ff9438d from peer 3b95f8c6589e473bb21db7681e227638 at {username='slave'} at 127.8.68.196:55885: session id = 3b95f8c6589e473bb21db7681e227638-8baf6913f38a42c3a369cd673ff9438d
I20240501 20:09:44.157943 12123 tablet_copy_source_session.cc:205] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.163738 13960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baf6913f38a42c3a369cd673ff9438d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.180645 13960 tablet_copy_client.cc:783] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.180958 13960 tablet_copy_client.cc:647] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.184005 13960 tablet_copy_client.cc:523] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:44.189684 13960 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: Bootstrap starting.
I20240501 20:09:44.209460 13971 tablet_copy_client.cc:308] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:09:44.216279 12103 consensus_queue.cc:237] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
W20240501 20:09:44.218924 12039 consensus_peers.cc:485] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351): Couldn't send request to peer 3b95f8c6589e473bb21db7681e227638. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d7dd13822784e789d558b4cc6c5010e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:44.219585 11970 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240501 20:09:44.221478 13754 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240501 20:09:44.226503 13754 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:44.226914 13960 log.cc:824] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:44.228513 11972 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:44.235882 11825 catalog_manager.cc:5484] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index -1 to 63, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:44.282747 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet 38b8e2865d134700ad0959372afc6670 from peer 463bd46a1f6e4e89a3040c4cf50dd919 ({username='slave'} at 127.8.68.198:55399)
I20240501 20:09:44.282954 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet 38b8e2865d134700ad0959372afc6670 from peer 463bd46a1f6e4e89a3040c4cf50dd919 at {username='slave'} at 127.8.68.198:55399: session id = 463bd46a1f6e4e89a3040c4cf50dd919-38b8e2865d134700ad0959372afc6670
I20240501 20:09:44.285915 12123 tablet_copy_source_session.cc:205] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.311204 13971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b8e2865d134700ad0959372afc6670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.336058 13971 tablet_copy_client.cc:783] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.336454 13971 tablet_copy_client.cc:647] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.338841 13971 tablet_copy_client.cc:523] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:44.344692 13985 ts_tablet_manager.cc:902] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:44.345580 13971 tablet_bootstrap.cc:492] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap starting.
I20240501 20:09:44.367369 13984 ts_tablet_manager.cc:902] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: Initiating tablet copy from peer 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:44.411883 13985 tablet_copy_client.cc:308] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:09:44.416226 13960 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:44.417001 13960 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: Bootstrap complete.
I20240501 20:09:44.417608 13960 ts_tablet_manager.cc:1367] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: Time spent bootstrapping tablet: real 0.228s	user 0.122s	sys 0.016s
I20240501 20:09:44.426877 13960 raft_consensus.cc:348] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:44.428416 13960 raft_consensus.cc:729] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b95f8c6589e473bb21db7681e227638, State: Initialized, Role: LEARNER
I20240501 20:09:44.429040 13960 consensus_queue.cc:260] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:44.438207 13960 ts_tablet_manager.cc:1398] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.007s
I20240501 20:09:44.440447 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 3b95f8c6589e473bb21db7681e227638-8baf6913f38a42c3a369cd673ff9438d received from {username='slave'} at 127.8.68.196:55885
I20240501 20:09:44.440582 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 3b95f8c6589e473bb21db7681e227638-8baf6913f38a42c3a369cd673ff9438d on tablet 8baf6913f38a42c3a369cd673ff9438d with peer 3b95f8c6589e473bb21db7681e227638
I20240501 20:09:44.452682 13971 log.cc:824] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:44.465724 13995 ts_tablet_manager.cc:902] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: Initiating tablet copy from peer 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:44.478484 12103 consensus_queue.cc:237] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:44.486543 13277 raft_consensus.cc:1203] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.56->[1.57-1.69]   Dedup: 1.69->[]
I20240501 20:09:44.490319 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet 575f6e5055d84dc38ef2326cec06fa47 from peer aed522b44a8e420daf65718450541c3c ({username='slave'} at 127.8.68.199:52135)
I20240501 20:09:44.490454 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet 575f6e5055d84dc38ef2326cec06fa47 from peer aed522b44a8e420daf65718450541c3c at {username='slave'} at 127.8.68.199:52135: session id = aed522b44a8e420daf65718450541c3c-575f6e5055d84dc38ef2326cec06fa47
I20240501 20:09:44.493330 12123 tablet_copy_source_session.cc:205] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.498061 13985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575f6e5055d84dc38ef2326cec06fa47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.509639 12233 raft_consensus.cc:1261] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 20:09:44.510779 13816 consensus_queue.cc:1035] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240501 20:09:44.516503 13816 raft_consensus.cc:2932] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index -1 to 98, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } } }
I20240501 20:09:44.518443 12233 raft_consensus.cc:2932] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index -1 to 98, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } } }
I20240501 20:09:44.523898 11825 catalog_manager.cc:5484] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index -1 to 98, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } } }
I20240501 20:09:44.546517 13985 tablet_copy_client.cc:783] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.546844 13985 tablet_copy_client.cc:647] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.549206 13985 tablet_copy_client.cc:523] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:44.553504 13985 tablet_bootstrap.cc:492] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: Bootstrap starting.
I20240501 20:09:44.556216 13995 tablet_copy_client.cc:308] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:42201
W20240501 20:09:44.560791 12036 consensus_peers.cc:485] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 535367472879403fa999dd00a155f042 (127.8.68.197:43403): Couldn't send request to peer 535367472879403fa999dd00a155f042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf6c676aa51e44a9b8a1c2a3c828ccd0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:44.567925 13984 tablet_copy_client.cc:308] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:42201
I20240501 20:09:44.578567 12253 tablet_copy_service.cc:140] P 18717c9754d04918815985cf07c1ddad: Received BeginTabletCopySession request for tablet d847215537454b4ba9e8a8b87ec91699 from peer 463bd46a1f6e4e89a3040c4cf50dd919 ({username='slave'} at 127.8.68.198:40347)
I20240501 20:09:44.578846 12253 tablet_copy_service.cc:161] P 18717c9754d04918815985cf07c1ddad: Beginning new tablet copy session on tablet d847215537454b4ba9e8a8b87ec91699 from peer 463bd46a1f6e4e89a3040c4cf50dd919 at {username='slave'} at 127.8.68.198:40347: session id = 463bd46a1f6e4e89a3040c4cf50dd919-d847215537454b4ba9e8a8b87ec91699
I20240501 20:09:44.582530 12253 tablet_copy_source_session.cc:205] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.586701 13995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d847215537454b4ba9e8a8b87ec91699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.589151 12233 consensus_queue.cc:237] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
W20240501 20:09:44.591295 12169 consensus_peers.cc:485] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad -> Peer 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198:35531): Couldn't send request to peer 463bd46a1f6e4e89a3040c4cf50dd919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ad26a211a9c4c609da559d2681c837a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:44.592104 11972 raft_consensus.cc:1261] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240501 20:09:44.592522 13995 tablet_copy_client.cc:783] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.592779 13995 tablet_copy_client.cc:647] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.593320 13985 log.cc:824] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:44.594249 13915 consensus_queue.cc:1035] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20240501 20:09:44.598148 13811 raft_consensus.cc:2932] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index -1 to 91, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } } }
I20240501 20:09:44.600093 11972 raft_consensus.cc:2932] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index -1 to 91, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } } }
I20240501 20:09:44.608186 11829 catalog_manager.cc:5484] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index -1 to 91, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:44.608613 13995 tablet_copy_client.cc:523] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:44.611701 13995 tablet_bootstrap.cc:492] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap starting.
I20240501 20:09:44.624608 12253 tablet_copy_service.cc:140] P 18717c9754d04918815985cf07c1ddad: Received BeginTabletCopySession request for tablet 0faf9b33582449ee9ffa353ae6ea3d6a from peer 535367472879403fa999dd00a155f042 ({username='slave'} at 127.8.68.197:34667)
I20240501 20:09:44.624787 12253 tablet_copy_service.cc:161] P 18717c9754d04918815985cf07c1ddad: Beginning new tablet copy session on tablet 0faf9b33582449ee9ffa353ae6ea3d6a from peer 535367472879403fa999dd00a155f042 at {username='slave'} at 127.8.68.197:34667: session id = 535367472879403fa999dd00a155f042-0faf9b33582449ee9ffa353ae6ea3d6a
I20240501 20:09:44.627485 12253 tablet_copy_source_session.cc:205] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.632596 13984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faf9b33582449ee9ffa353ae6ea3d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.639493 14015 ts_tablet_manager.cc:902] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:44.677651 14015 tablet_copy_client.cc:308] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:09:44.678372 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet f25dcb94e89c4c42979fdb39523b7fd2 from peer aed522b44a8e420daf65718450541c3c ({username='slave'} at 127.8.68.199:52135)
I20240501 20:09:44.678552 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet f25dcb94e89c4c42979fdb39523b7fd2 from peer aed522b44a8e420daf65718450541c3c at {username='slave'} at 127.8.68.199:52135: session id = aed522b44a8e420daf65718450541c3c-f25dcb94e89c4c42979fdb39523b7fd2
I20240501 20:09:44.681356 12123 tablet_copy_source_session.cc:205] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.695356 14015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25dcb94e89c4c42979fdb39523b7fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.708524 14015 tablet_copy_client.cc:783] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.708904 14015 tablet_copy_client.cc:647] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.711115 14015 tablet_copy_client.cc:523] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:44.714210 13960 ts_tablet_manager.cc:902] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:44.720254 13984 tablet_copy_client.cc:783] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.720592 13984 tablet_copy_client.cc:647] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.723536 14015 tablet_bootstrap.cc:492] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: Bootstrap starting.
I20240501 20:09:44.723670 12983 raft_consensus.cc:1050] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 3b95f8c6589e473bb21db7681e227638 to VOTER
I20240501 20:09:44.724416 12983 consensus_queue.cc:237] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:44.726702 13276 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240501 20:09:44.727699 13754 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240501 20:09:44.737080 12233 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240501 20:09:44.740778 13950 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index 57 to 71, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:44.741197 13277 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 57 to 71, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:44.747529 13984 tablet_copy_client.cc:523] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:44.749300 11829 catalog_manager.cc:5484] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 57 to 71, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:44.752460 13984 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: Bootstrap starting.
I20240501 20:09:44.754112 12103 consensus_queue.cc:237] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:44.760771 13276 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 20:09:44.761415 12983 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240501 20:09:44.763913 13816 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 71 to 72, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:44.764568 13277 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 71 to 72, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:44.769024 11829 catalog_manager.cc:5484] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 reported cstate change: config changed from index 71 to 72, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:44.771639 11818 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8baf6913f38a42c3a369cd673ff9438d with cas_config_opid_index 71: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:44.777487 13960 tablet_copy_client.cc:308] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:09:44.779018 12212 tablet_service.cc:1514] Processing DeleteTablet for tablet 8baf6913f38a42c3a369cd673ff9438d with delete_type TABLET_DATA_TOMBSTONED (TS 18717c9754d04918815985cf07c1ddad not found in new config with opid_index 72) from {username='slave'} at 127.0.0.1:50996
I20240501 20:09:44.779901 14034 tablet_replica.cc:330] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad: stopping tablet replica
I20240501 20:09:44.780316 14034 raft_consensus.cc:2229] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:44.780614 14034 raft_consensus.cc:2258] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:44.780918 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet 9d7dd13822784e789d558b4cc6c5010e from peer 3b95f8c6589e473bb21db7681e227638 ({username='slave'} at 127.8.68.196:55885)
I20240501 20:09:44.781066 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet 9d7dd13822784e789d558b4cc6c5010e from peer 3b95f8c6589e473bb21db7681e227638 at {username='slave'} at 127.8.68.196:55885: session id = 3b95f8c6589e473bb21db7681e227638-9d7dd13822784e789d558b4cc6c5010e
I20240501 20:09:44.781905 13985 tablet_bootstrap.cc:492] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=0 applied=59 ignored=0} inserts{seen=731 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:44.782907 14034 ts_tablet_manager.cc:1875] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:44.783756 12123 tablet_copy_source_session.cc:205] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.785138 13960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d7dd13822784e789d558b4cc6c5010e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.789201 13985 tablet_bootstrap.cc:492] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: Bootstrap complete.
I20240501 20:09:44.789674 13985 ts_tablet_manager.cc:1367] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: Time spent bootstrapping tablet: real 0.236s	user 0.116s	sys 0.009s
I20240501 20:09:44.801968 13985 raft_consensus.cc:348] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:44.802441 13985 raft_consensus.cc:729] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aed522b44a8e420daf65718450541c3c, State: Initialized, Role: LEARNER
I20240501 20:09:44.803022 13985 consensus_queue.cc:260] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 59, Last appended: 1.59, Last appended by leader: 59, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:44.851943 13984 log.cc:824] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:44.883167 13985 ts_tablet_manager.cc:1398] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c: Time spent starting tablet: real 0.093s	user 0.017s	sys 0.014s
I20240501 20:09:44.883977 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session aed522b44a8e420daf65718450541c3c-575f6e5055d84dc38ef2326cec06fa47 received from {username='slave'} at 127.8.68.199:52135
I20240501 20:09:44.884133 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session aed522b44a8e420daf65718450541c3c-575f6e5055d84dc38ef2326cec06fa47 on tablet 575f6e5055d84dc38ef2326cec06fa47 with peer aed522b44a8e420daf65718450541c3c
I20240501 20:09:44.900792 13971 tablet_bootstrap.cc:492] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap replayed 1/1 log segments. Stats: ops{read=77 overwritten=0 applied=77 ignored=0} inserts{seen=708 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:44.901371 13971 tablet_bootstrap.cc:492] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap complete.
I20240501 20:09:44.901832 13971 ts_tablet_manager.cc:1367] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: Time spent bootstrapping tablet: real 0.558s	user 0.129s	sys 0.019s
I20240501 20:09:44.903209 13960 tablet_copy_client.cc:783] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.903534 13960 tablet_copy_client.cc:647] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.911964 13960 tablet_copy_client.cc:523] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:44.914266 13971 raft_consensus.cc:348] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:44.914814 13971 raft_consensus.cc:729] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 463bd46a1f6e4e89a3040c4cf50dd919, State: Initialized, Role: LEARNER
I20240501 20:09:44.917766 14034 ts_tablet_manager.cc:1888] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.70
I20240501 20:09:44.915409 13971 consensus_queue.cc:260] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 77, Last appended: 1.77, Last appended by leader: 77, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:44.936309 14041 ts_tablet_manager.cc:902] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:44.938746 13971 ts_tablet_manager.cc:1398] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919: Time spent starting tablet: real 0.037s	user 0.003s	sys 0.012s
I20240501 20:09:44.941121 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 463bd46a1f6e4e89a3040c4cf50dd919-38b8e2865d134700ad0959372afc6670 received from {username='slave'} at 127.8.68.198:55399
I20240501 20:09:44.941265 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 463bd46a1f6e4e89a3040c4cf50dd919-38b8e2865d134700ad0959372afc6670 on tablet 38b8e2865d134700ad0959372afc6670 with peer 463bd46a1f6e4e89a3040c4cf50dd919
I20240501 20:09:44.991837 14034 log.cc:1196] T 8baf6913f38a42c3a369cd673ff9438d P 18717c9754d04918815985cf07c1ddad: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/8baf6913f38a42c3a369cd673ff9438d
I20240501 20:09:44.995275 13960 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: Bootstrap starting.
I20240501 20:09:44.998653 11816 catalog_manager.cc:4830] TS 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201): tablet 8baf6913f38a42c3a369cd673ff9438d (table rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]) successfully deleted
I20240501 20:09:45.011126 14015 tablet_bootstrap.cc:492] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=89 ignored=0} inserts{seen=720 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:09:45.011786 14015 tablet_bootstrap.cc:492] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: Bootstrap complete.
I20240501 20:09:45.012107 14015 ts_tablet_manager.cc:1367] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: Time spent bootstrapping tablet: real 0.289s	user 0.130s	sys 0.011s
I20240501 20:09:45.013288 14015 raft_consensus.cc:348] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:45.043840 14015 raft_consensus.cc:729] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aed522b44a8e420daf65718450541c3c, State: Initialized, Role: LEARNER
I20240501 20:09:45.044135 14015 consensus_queue.cc:260] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:45.045048 14015 ts_tablet_manager.cc:1398] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c: Time spent starting tablet: real 0.033s	user 0.025s	sys 0.005s
I20240501 20:09:45.045740 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session aed522b44a8e420daf65718450541c3c-f25dcb94e89c4c42979fdb39523b7fd2 received from {username='slave'} at 127.8.68.199:52135
I20240501 20:09:45.045878 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session aed522b44a8e420daf65718450541c3c-f25dcb94e89c4c42979fdb39523b7fd2 on tablet f25dcb94e89c4c42979fdb39523b7fd2 with peer aed522b44a8e420daf65718450541c3c
I20240501 20:09:45.049371 13669 raft_consensus.cc:1203] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.71->[1.72-1.96]   Dedup: 1.90->[1.91-1.96]
I20240501 20:09:45.079857 11972 consensus_queue.cc:237] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:45.110934 13971 ts_tablet_manager.cc:902] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: Initiating tablet copy from peer 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:45.114823 14041 tablet_copy_client.cc:308] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:09:45.116567 12233 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240501 20:09:45.117650 14051 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240501 20:09:45.129451 13162 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } } }
I20240501 20:09:45.138959 13782 raft_consensus.cc:1050] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER aed522b44a8e420daf65718450541c3c to VOTER
I20240501 20:09:45.139755 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet cf6c676aa51e44a9b8a1c2a3c828ccd0 from peer 535367472879403fa999dd00a155f042 ({username='slave'} at 127.8.68.197:35537)
I20240501 20:09:45.139918 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet cf6c676aa51e44a9b8a1c2a3c828ccd0 from peer 535367472879403fa999dd00a155f042 at {username='slave'} at 127.8.68.197:35537: session id = 535367472879403fa999dd00a155f042-cf6c676aa51e44a9b8a1c2a3c828ccd0
I20240501 20:09:45.139875 13782 consensus_queue.cc:237] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:45.142227 13669 raft_consensus.cc:1261] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 20:09:45.142915 12123 tablet_copy_source_session.cc:205] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:45.146777 14041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6c676aa51e44a9b8a1c2a3c828ccd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.149163 13950 consensus_queue.cc:1035] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 20:09:45.155715 11829 catalog_manager.cc:5484] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index -1 to 84, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:45.157440 13782 raft_consensus.cc:2932] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 72 to 99, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:45.159525 12233 raft_consensus.cc:1261] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 20:09:45.162812 14041 tablet_copy_client.cc:783] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:45.163075 14041 tablet_copy_client.cc:647] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:45.163084 11829 catalog_manager.cc:5484] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 72 to 99, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:45.167713 14041 tablet_copy_client.cc:523] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:45.168169 13782 consensus_queue.cc:1035] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 20:09:45.168993 13669 raft_consensus.cc:2932] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 72 to 99, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:45.172022 12103 consensus_queue.cc:237] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:45.176112 14041 tablet_bootstrap.cc:492] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: Bootstrap starting.
I20240501 20:09:45.176852 13668 raft_consensus.cc:1261] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240501 20:09:45.177088 12233 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } } }
I20240501 20:09:45.178133 12229 raft_consensus.cc:2932] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 72 to 99, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:45.192016 13984 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: Bootstrap replayed 1/1 log segments. Stats: ops{read=102 overwritten=0 applied=102 ignored=0} inserts{seen=774 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:45.193949 13984 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: Bootstrap complete.
I20240501 20:09:45.195153 13984 ts_tablet_manager.cc:1367] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: Time spent bootstrapping tablet: real 0.443s	user 0.173s	sys 0.001s
I20240501 20:09:45.203858 13971 tablet_copy_client.cc:308] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:42201
I20240501 20:09:45.205009 12253 tablet_copy_service.cc:140] P 18717c9754d04918815985cf07c1ddad: Received BeginTabletCopySession request for tablet 4ad26a211a9c4c609da559d2681c837a from peer 463bd46a1f6e4e89a3040c4cf50dd919 ({username='slave'} at 127.8.68.198:40347)
I20240501 20:09:45.205163 12253 tablet_copy_service.cc:161] P 18717c9754d04918815985cf07c1ddad: Beginning new tablet copy session on tablet 4ad26a211a9c4c609da559d2681c837a from peer 463bd46a1f6e4e89a3040c4cf50dd919 at {username='slave'} at 127.8.68.198:40347: session id = 463bd46a1f6e4e89a3040c4cf50dd919-4ad26a211a9c4c609da559d2681c837a
I20240501 20:09:45.207947 12253 tablet_copy_source_session.cc:205] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:45.207811 13984 raft_consensus.cc:348] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:45.208493 13984 raft_consensus.cc:729] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 535367472879403fa999dd00a155f042, State: Initialized, Role: LEARNER
I20240501 20:09:45.209137 13984 consensus_queue.cc:260] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 1.102, Last appended by leader: 102, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:45.212837 13971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad26a211a9c4c609da559d2681c837a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.218904 13971 tablet_copy_client.cc:783] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:45.219210 13971 tablet_copy_client.cc:647] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:45.227391 13984 ts_tablet_manager.cc:1398] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042: Time spent starting tablet: real 0.032s	user 0.010s	sys 0.003s
I20240501 20:09:45.230046 12253 tablet_copy_service.cc:342] P 18717c9754d04918815985cf07c1ddad: Request end of tablet copy session 535367472879403fa999dd00a155f042-0faf9b33582449ee9ffa353ae6ea3d6a received from {username='slave'} at 127.8.68.197:34667
I20240501 20:09:45.230198 12253 tablet_copy_service.cc:433] P 18717c9754d04918815985cf07c1ddad: ending tablet copy session 535367472879403fa999dd00a155f042-0faf9b33582449ee9ffa353ae6ea3d6a on tablet 0faf9b33582449ee9ffa353ae6ea3d6a with peer 535367472879403fa999dd00a155f042
I20240501 20:09:45.231108 13971 tablet_copy_client.cc:523] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:45.234839 13971 tablet_bootstrap.cc:492] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap starting.
W20240501 20:09:45.257372 11906 consensus_peers.cc:485] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 -> Peer aed522b44a8e420daf65718450541c3c (127.8.68.199:40105): Couldn't send request to peer aed522b44a8e420daf65718450541c3c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8f9c24067b442299db6fa5831feb56d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:45.268801 12103 consensus_queue.cc:237] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
W20240501 20:09:45.271579 12039 consensus_peers.cc:485] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351): Couldn't send request to peer 3b95f8c6589e473bb21db7681e227638. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7eb828d8511f40c78004d0aaba843df5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:45.280450 13782 consensus_queue.cc:1035] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240501 20:09:45.281453 11972 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240501 20:09:45.293573 13782 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240501 20:09:45.313758 13816 raft_consensus.cc:2932] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index 99 to 100, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:45.332686 11818 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f25dcb94e89c4c42979fdb39523b7fd2 with cas_config_opid_index 99: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:45.335093 11825 catalog_manager.cc:5484] T f25dcb94e89c4c42979fdb39523b7fd2 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 99 to 100, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:45.335037 13816 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:45.338379 13668 raft_consensus.cc:2932] T f25dcb94e89c4c42979fdb39523b7fd2 P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index 99 to 100, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:45.340734 11973 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:45.341940 11825 catalog_manager.cc:5484] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index -1 to 71, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:45.347159 12212 tablet_service.cc:1514] Processing DeleteTablet for tablet f25dcb94e89c4c42979fdb39523b7fd2 with delete_type TABLET_DATA_TOMBSTONED (TS 18717c9754d04918815985cf07c1ddad not found in new config with opid_index 100) from {username='slave'} at 127.0.0.1:50996
I20240501 20:09:45.347473 14034 tablet_replica.cc:330] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad: stopping tablet replica
I20240501 20:09:45.347666 14034 raft_consensus.cc:2229] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:45.348423 14034 raft_consensus.cc:2258] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:45.348526 13408 raft_consensus.cc:1203] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.80->[1.81-1.110]   Dedup: 1.102->[1.103-1.110]
I20240501 20:09:45.349607 14034 ts_tablet_manager.cc:1875] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:45.353140 13539 raft_consensus.cc:1203] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.58->[1.59-1.91]   Dedup: 1.77->[1.78-1.91]
I20240501 20:09:45.356693 14034 ts_tablet_manager.cc:1888] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.99
I20240501 20:09:45.356803 14034 log.cc:1196] T f25dcb94e89c4c42979fdb39523b7fd2 P 18717c9754d04918815985cf07c1ddad: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/f25dcb94e89c4c42979fdb39523b7fd2
I20240501 20:09:45.357482 11816 catalog_manager.cc:4830] TS 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201): tablet f25dcb94e89c4c42979fdb39523b7fd2 (table rebalance_test_table_2 [id=b40a568bc0e8467881e6ef108d91304f]) successfully deleted
I20240501 20:09:45.361572 13668 raft_consensus.cc:1203] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.50->[1.51-1.68]   Dedup: 1.59->[1.60-1.68]
I20240501 20:09:45.461149 12103 consensus_queue.cc:237] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:45.479600 13960 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: Bootstrap replayed 1/1 log segments. Stats: ops{read=68 overwritten=0 applied=68 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:45.480079 13960 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: Bootstrap complete.
I20240501 20:09:45.480381 13960 ts_tablet_manager.cc:1367] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: Time spent bootstrapping tablet: real 0.485s	user 0.110s	sys 0.011s
I20240501 20:09:45.481343 13960 raft_consensus.cc:348] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:45.481617 13960 raft_consensus.cc:729] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b95f8c6589e473bb21db7681e227638, State: Initialized, Role: LEARNER
I20240501 20:09:45.481858 13960 consensus_queue.cc:260] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 68, Last appended: 1.68, Last appended by leader: 68, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:45.490923 13960 ts_tablet_manager.cc:1398] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.001s
I20240501 20:09:45.492462 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 3b95f8c6589e473bb21db7681e227638-9d7dd13822784e789d558b4cc6c5010e received from {username='slave'} at 127.8.68.196:55885
I20240501 20:09:45.492600 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 3b95f8c6589e473bb21db7681e227638-9d7dd13822784e789d558b4cc6c5010e on tablet 9d7dd13822784e789d558b4cc6c5010e with peer 3b95f8c6589e473bb21db7681e227638
W20240501 20:09:45.498397 12036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:40105, user_credentials={real_user=slave}} blocked reactor thread for 61425.1us
I20240501 20:09:45.509362 13811 raft_consensus.cc:1050] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: attempting to promote NON_VOTER 535367472879403fa999dd00a155f042 to VOTER
I20240501 20:09:45.515022 11972 raft_consensus.cc:1261] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
W20240501 20:09:45.516986 12036 consensus_peers.cc:485] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 535367472879403fa999dd00a155f042 (127.8.68.197:43403): Couldn't send request to peer 535367472879403fa999dd00a155f042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67043e58e5fd4233bc945432eeae1227. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:45.518322 13818 consensus_queue.cc:1035] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240501 20:09:45.525362 13811 consensus_queue.cc:237] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:45.527616 13818 raft_consensus.cc:1050] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER aed522b44a8e420daf65718450541c3c to VOTER
I20240501 20:09:45.529917 13754 raft_consensus.cc:1050] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 to VOTER
I20240501 20:09:45.530616 13754 consensus_queue.cc:237] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:45.533098 13539 raft_consensus.cc:1261] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 20:09:45.533331 13818 consensus_queue.cc:237] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:45.534204 13408 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240501 20:09:45.534976 13818 consensus_queue.cc:1035] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s
I20240501 20:09:45.537955 13950 raft_consensus.cc:2932] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 59 to 93, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.539072 13811 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.003s
I20240501 20:09:45.542487 13539 raft_consensus.cc:2932] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 59 to 93, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.543347 11973 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240501 20:09:45.544440 14078 raft_consensus.cc:2932] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index -1 to 100, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } } }
I20240501 20:09:45.548792 11829 catalog_manager.cc:5484] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 reported cstate change: config changed from index 59 to 93, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.549659 11973 raft_consensus.cc:1261] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240501 20:09:45.550377 13669 raft_consensus.cc:1261] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240501 20:09:45.552350 12229 raft_consensus.cc:1261] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 20:09:45.552435 13975 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20240501 20:09:45.552862 13975 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index 81 to 113, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:45.554415 13408 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index 81 to 113, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:45.557670 11825 catalog_manager.cc:5484] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index 81 to 113, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:45.563621 11972 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index 81 to 113, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:45.569144 12229 consensus_queue.cc:237] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 1.113, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:45.572175 13408 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240501 20:09:45.574846 13811 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240501 20:09:45.578178 13802 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.114: config changed from index 113 to 114, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:45.582592 14041 tablet_bootstrap.cc:492] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: Bootstrap replayed 1/1 log segments. Stats: ops{read=108 overwritten=0 applied=108 ignored=0} inserts{seen=739 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:45.583065 14041 tablet_bootstrap.cc:492] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: Bootstrap complete.
I20240501 20:09:45.583345 14041 ts_tablet_manager.cc:1367] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: Time spent bootstrapping tablet: real 0.408s	user 0.141s	sys 0.022s
I20240501 20:09:45.584265 14041 raft_consensus.cc:348] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:45.584494 14041 raft_consensus.cc:729] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 535367472879403fa999dd00a155f042, State: Initialized, Role: LEARNER
I20240501 20:09:45.584697 14041 consensus_queue.cc:260] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 108, Last appended: 1.108, Last appended by leader: 108, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:45.584987 11829 catalog_manager.cc:5484] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index 113 to 114, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:45.585394 14041 ts_tablet_manager.cc:1398] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:45.586078 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 535367472879403fa999dd00a155f042-cf6c676aa51e44a9b8a1c2a3c828ccd0 received from {username='slave'} at 127.8.68.197:35537
I20240501 20:09:45.586210 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 535367472879403fa999dd00a155f042-cf6c676aa51e44a9b8a1c2a3c828ccd0 on tablet cf6c676aa51e44a9b8a1c2a3c828ccd0 with peer 535367472879403fa999dd00a155f042
I20240501 20:09:45.588802 13276 raft_consensus.cc:1203] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.62->[1.63-1.75]   Dedup: 1.68->[1.69-1.75]
I20240501 20:09:45.589887 11952 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faf9b33582449ee9ffa353ae6ea3d6a with delete_type TABLET_DATA_TOMBSTONED (TS dc71b180f72b4c4697d28b13b363d705 not found in new config with opid_index 114) from {username='slave'} at 127.0.0.1:58526
I20240501 20:09:45.595046 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0faf9b33582449ee9ffa353ae6ea3d6a with cas_config_opid_index 113: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:45.595507 13408 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 113 to 114, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:45.597760 13971 tablet_bootstrap.cc:492] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=0 applied=99 ignored=0} inserts{seen=784 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:45.598209 13971 tablet_bootstrap.cc:492] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap complete.
I20240501 20:09:45.598534 13971 ts_tablet_manager.cc:1367] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: Time spent bootstrapping tablet: real 0.364s	user 0.146s	sys 0.015s
I20240501 20:09:45.599478 13971 raft_consensus.cc:348] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:45.599751 13971 raft_consensus.cc:729] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 463bd46a1f6e4e89a3040c4cf50dd919, State: Initialized, Role: LEARNER
I20240501 20:09:45.599960 13971 consensus_queue.cc:260] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 1.99, Last appended by leader: 99, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:45.600714 13971 ts_tablet_manager.cc:1398] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:45.602025 12253 tablet_copy_service.cc:342] P 18717c9754d04918815985cf07c1ddad: Request end of tablet copy session 463bd46a1f6e4e89a3040c4cf50dd919-4ad26a211a9c4c609da559d2681c837a received from {username='slave'} at 127.8.68.198:40347
I20240501 20:09:45.602166 12253 tablet_copy_service.cc:433] P 18717c9754d04918815985cf07c1ddad: ending tablet copy session 463bd46a1f6e4e89a3040c4cf50dd919-4ad26a211a9c4c609da559d2681c837a on tablet 4ad26a211a9c4c609da559d2681c837a with peer 463bd46a1f6e4e89a3040c4cf50dd919
W20240501 20:09:45.603122 12036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:42201, user_credentials={real_user=slave}} blocked reactor thread for 51785.4us
I20240501 20:09:45.604380 13782 consensus_queue.cc:1035] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s
I20240501 20:09:45.604808 13782 consensus_queue.cc:1035] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240501 20:09:45.605320 14078 consensus_queue.cc:1035] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.001s
I20240501 20:09:45.609949 11827 catalog_manager.cc:5484] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index -1 to 100, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:45.613041 14089 tablet_replica.cc:330] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705: stopping tablet replica
I20240501 20:09:45.613479 14089 raft_consensus.cc:2229] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:45.614502 14089 raft_consensus.cc:2258] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:45.630196 14089 ts_tablet_manager.cc:1875] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:45.630789 11973 raft_consensus.cc:2932] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index -1 to 100, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } } }
I20240501 20:09:45.632405 13995 tablet_bootstrap.cc:492] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=80 ignored=0} inserts{seen=752 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:45.632777 13995 tablet_bootstrap.cc:492] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap complete.
I20240501 20:09:45.633029 13995 ts_tablet_manager.cc:1367] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: Time spent bootstrapping tablet: real 1.021s	user 0.116s	sys 0.023s
I20240501 20:09:45.633733 13995 raft_consensus.cc:348] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:45.633950 13995 raft_consensus.cc:729] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 463bd46a1f6e4e89a3040c4cf50dd919, State: Initialized, Role: LEARNER
I20240501 20:09:45.634150 13995 consensus_queue.cc:260] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:45.638576 14089 ts_tablet_manager.cc:1888] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.113
I20240501 20:09:45.638859 14089 log.cc:1196] T 0faf9b33582449ee9ffa353ae6ea3d6a P dc71b180f72b4c4697d28b13b363d705: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/0faf9b33582449ee9ffa353ae6ea3d6a
I20240501 20:09:45.638943 13995 ts_tablet_manager.cc:1398] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 20:09:45.640799 12253 tablet_copy_service.cc:342] P 18717c9754d04918815985cf07c1ddad: Request end of tablet copy session 463bd46a1f6e4e89a3040c4cf50dd919-d847215537454b4ba9e8a8b87ec91699 received from {username='slave'} at 127.8.68.198:40347
I20240501 20:09:45.640966 12253 tablet_copy_service.cc:433] P 18717c9754d04918815985cf07c1ddad: ending tablet copy session 463bd46a1f6e4e89a3040c4cf50dd919-d847215537454b4ba9e8a8b87ec91699 on tablet d847215537454b4ba9e8a8b87ec91699 with peer 463bd46a1f6e4e89a3040c4cf50dd919
I20240501 20:09:45.641736 11816 catalog_manager.cc:4830] TS dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397): tablet 0faf9b33582449ee9ffa353ae6ea3d6a (table rebalance_test_table_2 [id=b40a568bc0e8467881e6ef108d91304f]) successfully deleted
I20240501 20:09:45.646596 13539 raft_consensus.cc:1203] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.90->[1.91-1.105]   Dedup: 1.99->[1.100-1.105]
W20240501 20:09:45.662113 12036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:40105, user_credentials={real_user=slave}} blocked reactor thread for 57882.1us
I20240501 20:09:45.663396 12233 raft_consensus.cc:2932] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 59 to 93, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.669236 14095 ts_tablet_manager.cc:902] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: Initiating tablet copy from peer dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:45.674458 13818 raft_consensus.cc:2932] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 51 to 70, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:45.679956 11827 catalog_manager.cc:5484] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 51 to 70, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:45.683135 13668 raft_consensus.cc:2932] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 51 to 70, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:45.738548 14095 tablet_copy_client.cc:308] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:39397
I20240501 20:09:45.761324 13950 raft_consensus.cc:1050] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 3b95f8c6589e473bb21db7681e227638 to VOTER
I20240501 20:09:45.761919 13950 consensus_queue.cc:237] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:45.763722 13277 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 20:09:45.764235 13816 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 20:09:45.767076 13818 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 63 to 80, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:45.770920 13277 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 63 to 80, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:45.774408 11827 catalog_manager.cc:5484] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 63 to 80, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:45.781100 13802 raft_consensus.cc:1050] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad: attempting to promote NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 to VOTER
I20240501 20:09:45.781610 13802 consensus_queue.cc:237] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:45.784566 13538 raft_consensus.cc:1261] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240501 20:09:45.785521 13815 consensus_queue.cc:1035] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240501 20:09:45.788295 13815 raft_consensus.cc:2932] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index 91 to 108, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.788686 13538 raft_consensus.cc:2932] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 91 to 108, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.793938 11827 catalog_manager.cc:5484] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919 reported cstate change: config changed from index 91 to 108, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.796032 11969 raft_consensus.cc:1261] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240501 20:09:45.802708 12233 consensus_queue.cc:237] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:45.804725 13539 raft_consensus.cc:1261] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240501 20:09:45.805249 13802 consensus_queue.cc:1035] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 108, Time since last communication: 0.000s
I20240501 20:09:45.807476 13975 raft_consensus.cc:2932] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index 108 to 110, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.810946 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4ad26a211a9c4c609da559d2681c837a with cas_config_opid_index 108: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:45.816834 13538 raft_consensus.cc:2932] T 4ad26a211a9c4c609da559d2681c837a P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 108 to 110, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.817292 11827 catalog_manager.cc:5484] T 4ad26a211a9c4c609da559d2681c837a P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index 108 to 110, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:45.826670 13539 raft_consensus.cc:1203] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.73->[1.74-1.92]   Dedup: 1.80->[1.81-1.92]
I20240501 20:09:45.828256 11952 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ad26a211a9c4c609da559d2681c837a with delete_type TABLET_DATA_TOMBSTONED (TS dc71b180f72b4c4697d28b13b363d705 not found in new config with opid_index 110) from {username='slave'} at 127.0.0.1:58526
I20240501 20:09:45.828599 14089 tablet_replica.cc:330] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: stopping tablet replica
I20240501 20:09:45.828799 14089 raft_consensus.cc:2229] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:45.828979 14089 pending_rounds.cc:62] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: Trying to abort 1 pending ops.
I20240501 20:09:45.829109 14089 pending_rounds.cc:68] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: Aborting op as it isn't in flight: id { term: 1 index: 107 } timestamp: 7022977784947843072 op_type: WRITE_OP write_request { tablet_id: "4ad26a211a9c4c609da559d2681c837a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\215LM^\367\200b\244\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""6a9b4a0b72345c0e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977784857702400 authz_token { token_data: "\010\260\305\312\261\006\"3\n\005slave\022*\n 801bc83cb7614cdaaf85134efd64f1d7\020\001\030\001 \001(\001" signature: "<redacted>""\223{\212\360\221\266\237O\r\200\034\320\'\270i\333UEr\244\257\307\223 U5\364\360\032y\010\276\373E\211k\257]\314\240Cb\373\367&-\232\247\211J\307\225H\332|G5U\3725lQ\"\313" signing_key_seq_num: 0 } } request_id { client_id: "1e58759763df4276893282603ef8ba61" seq_no: 336 first_incomplete_seq_no: 336 attempt_no: 0 }
I20240501 20:09:45.829782 14089 raft_consensus.cc:2258] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:45.831249 14089 ts_tablet_manager.cc:1875] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:45.837651 14089 ts_tablet_manager.cc:1888] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.107
I20240501 20:09:45.837841 14089 log.cc:1196] T 4ad26a211a9c4c609da559d2681c837a P dc71b180f72b4c4697d28b13b363d705: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ad26a211a9c4c609da559d2681c837a
I20240501 20:09:45.839116 11816 catalog_manager.cc:4830] TS dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397): tablet 4ad26a211a9c4c609da559d2681c837a (table rebalance_test_table_0 [id=801bc83cb7614cdaaf85134efd64f1d7]) successfully deleted
I20240501 20:09:45.839175 11973 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 20:09:45.848573 14078 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 20:09:45.860024 11969 raft_consensus.cc:2932] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 51 to 70, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:45.870733 13802 raft_consensus.cc:1050] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad: attempting to promote NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 to VOTER
I20240501 20:09:45.870767 11969 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 63 to 80, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:45.871363 13802 consensus_queue.cc:237] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:45.873255 13539 raft_consensus.cc:1261] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 LEARNER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240501 20:09:45.874405 13915 consensus_queue.cc:1035] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240501 20:09:45.876926 11993 tablet_copy_service.cc:140] P dc71b180f72b4c4697d28b13b363d705: Received BeginTabletCopySession request for tablet f8f9c24067b442299db6fa5831feb56d from peer aed522b44a8e420daf65718450541c3c ({username='slave'} at 127.8.68.199:47273)
I20240501 20:09:45.877184 11993 tablet_copy_service.cc:161] P dc71b180f72b4c4697d28b13b363d705: Beginning new tablet copy session on tablet f8f9c24067b442299db6fa5831feb56d from peer aed522b44a8e420daf65718450541c3c at {username='slave'} at 127.8.68.199:47273: session id = aed522b44a8e420daf65718450541c3c-f8f9c24067b442299db6fa5831feb56d
I20240501 20:09:45.878439 13960 ts_tablet_manager.cc:902] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:45.879699 11969 raft_consensus.cc:1261] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240501 20:09:45.883527 13915 consensus_queue.cc:1035] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240501 20:09:45.884984 11993 tablet_copy_source_session.cc:205] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:45.888545 13811 raft_consensus.cc:2932] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index 74 to 94, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.903120 11827 catalog_manager.cc:5484] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index 74 to 94, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:45.921607 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f9c24067b442299db6fa5831feb56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.893644 11973 raft_consensus.cc:2932] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 74 to 94, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.930433 14095 tablet_copy_client.cc:783] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:45.930671 14095 tablet_copy_client.cc:647] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:45.933499 14095 tablet_copy_client.cc:523] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:45.937713 13539 raft_consensus.cc:2932] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 74 to 94, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.938369 14095 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: Bootstrap starting.
I20240501 20:09:45.953030 12233 consensus_queue.cc:237] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:45.954623 13539 raft_consensus.cc:1261] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 20:09:45.955394 13975 consensus_queue.cc:1035] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 20:09:45.964643 13408 raft_consensus.cc:1203] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.97->[1.98-1.117]   Dedup: 1.108->[1.109-1.117]
I20240501 20:09:45.965163 13975 raft_consensus.cc:2932] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index 94 to 95, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.969584 13537 raft_consensus.cc:2932] T d847215537454b4ba9e8a8b87ec91699 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 94 to 95, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:45.974900 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d847215537454b4ba9e8a8b87ec91699 with cas_config_opid_index 94: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:45.976781 11828 catalog_manager.cc:5484] T d847215537454b4ba9e8a8b87ec91699 P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index 94 to 95, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:45.978919 14041 ts_tablet_manager.cc:902] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:45.981559 11952 tablet_service.cc:1514] Processing DeleteTablet for tablet d847215537454b4ba9e8a8b87ec91699 with delete_type TABLET_DATA_TOMBSTONED (TS dc71b180f72b4c4697d28b13b363d705 not found in new config with opid_index 95) from {username='slave'} at 127.0.0.1:58526
I20240501 20:09:45.981926 14089 tablet_replica.cc:330] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705: stopping tablet replica
I20240501 20:09:45.982136 14089 raft_consensus.cc:2229] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:45.982784 14089 raft_consensus.cc:2258] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:45.984140 14089 ts_tablet_manager.cc:1875] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:46.018249 14089 ts_tablet_manager.cc:1888] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.94
I20240501 20:09:46.018646 14089 log.cc:1196] T d847215537454b4ba9e8a8b87ec91699 P dc71b180f72b4c4697d28b13b363d705: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/d847215537454b4ba9e8a8b87ec91699
I20240501 20:09:46.019526 11816 catalog_manager.cc:4830] TS dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397): tablet d847215537454b4ba9e8a8b87ec91699 (table rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]) successfully deleted
I20240501 20:09:46.050542 11969 consensus_queue.cc:237] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 1.128, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:46.052942 12229 raft_consensus.cc:1261] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
W20240501 20:09:46.053279 11906 consensus_peers.cc:485] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 -> Peer aed522b44a8e420daf65718450541c3c (127.8.68.199:40105): Couldn't send request to peer aed522b44a8e420daf65718450541c3c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c0a1865e31241569216a3b4b072829c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:46.053689 14101 consensus_queue.cc:1035] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20240501 20:09:46.055936 14051 raft_consensus.cc:2932] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.129: config changed from index -1 to 129, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } } }
I20240501 20:09:46.061977 11827 catalog_manager.cc:5484] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index -1 to 129, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:46.065788 12229 raft_consensus.cc:2932] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index -1 to 129, NON_VOTER aed522b44a8e420daf65718450541c3c (127.8.68.199) added. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } } }
I20240501 20:09:46.080407 13960 tablet_copy_client.cc:308] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:09:46.081091 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet 7eb828d8511f40c78004d0aaba843df5 from peer 3b95f8c6589e473bb21db7681e227638 ({username='slave'} at 127.8.68.196:55885)
I20240501 20:09:46.081246 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet 7eb828d8511f40c78004d0aaba843df5 from peer 3b95f8c6589e473bb21db7681e227638 at {username='slave'} at 127.8.68.196:55885: session id = 3b95f8c6589e473bb21db7681e227638-7eb828d8511f40c78004d0aaba843df5
I20240501 20:09:46.083997 12123 tablet_copy_source_session.cc:205] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:46.085346 13960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb828d8511f40c78004d0aaba843df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.086522 14041 tablet_copy_client.cc:308] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:09:46.087445 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet 67043e58e5fd4233bc945432eeae1227 from peer 535367472879403fa999dd00a155f042 ({username='slave'} at 127.8.68.197:35537)
I20240501 20:09:46.087644 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet 67043e58e5fd4233bc945432eeae1227 from peer 535367472879403fa999dd00a155f042 at {username='slave'} at 127.8.68.197:35537: session id = 535367472879403fa999dd00a155f042-67043e58e5fd4233bc945432eeae1227
I20240501 20:09:46.094906 12123 tablet_copy_source_session.cc:205] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:46.096174 14041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67043e58e5fd4233bc945432eeae1227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.105484 13960 tablet_copy_client.cc:783] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:46.105717 13960 tablet_copy_client.cc:647] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:46.108031 13960 tablet_copy_client.cc:523] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:46.110208 14041 tablet_copy_client.cc:783] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:46.110483 14041 tablet_copy_client.cc:647] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:46.114105 14041 tablet_copy_client.cc:523] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:46.117323 14041 tablet_bootstrap.cc:492] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: Bootstrap starting.
I20240501 20:09:46.121129 13960 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: Bootstrap starting.
I20240501 20:09:46.167649 14078 raft_consensus.cc:1050] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 535367472879403fa999dd00a155f042 to VOTER
I20240501 20:09:46.168308 14078 consensus_queue.cc:237] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:46.170845 13408 raft_consensus.cc:1261] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240501 20:09:46.174005 12229 raft_consensus.cc:1261] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240501 20:09:46.174685 11970 consensus_queue.cc:237] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:46.175156 14092 consensus_queue.cc:1035] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20240501 20:09:46.205268 13818 consensus_queue.cc:1035] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20240501 20:09:46.210076 13754 raft_consensus.cc:2932] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index 98 to 123, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:46.213284 12233 raft_consensus.cc:2932] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 98 to 123, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:46.219192 11827 catalog_manager.cc:5484] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 98 to 123, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:46.448421 13960 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: Bootstrap replayed 1/1 log segments. Stats: ops{read=79 overwritten=0 applied=79 ignored=0} inserts{seen=764 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:46.448889 13960 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: Bootstrap complete.
I20240501 20:09:46.451393 13960 ts_tablet_manager.cc:1367] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: Time spent bootstrapping tablet: real 0.330s	user 0.121s	sys 0.020s
I20240501 20:09:46.453552 13960 raft_consensus.cc:348] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:46.454607 13960 raft_consensus.cc:729] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b95f8c6589e473bb21db7681e227638, State: Initialized, Role: LEARNER
I20240501 20:09:46.455008 13960 consensus_queue.cc:260] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 1.79, Last appended by leader: 79, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:46.461735 12103 consensus_queue.cc:237] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 1.129, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:46.462306 13407 raft_consensus.cc:2932] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 98 to 123, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:46.473076 13960 ts_tablet_manager.cc:1398] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.000s
I20240501 20:09:46.474892 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 3b95f8c6589e473bb21db7681e227638-7eb828d8511f40c78004d0aaba843df5 received from {username='slave'} at 127.8.68.196:55885
I20240501 20:09:46.475018 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 3b95f8c6589e473bb21db7681e227638-7eb828d8511f40c78004d0aaba843df5 on tablet 7eb828d8511f40c78004d0aaba843df5 with peer 3b95f8c6589e473bb21db7681e227638
I20240501 20:09:46.522023 14041 tablet_bootstrap.cc:492] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: Bootstrap replayed 1/1 log segments. Stats: ops{read=103 overwritten=0 applied=102 ignored=0} inserts{seen=774 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:09:46.522727 14041 tablet_bootstrap.cc:492] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: Bootstrap complete.
I20240501 20:09:46.525249 14041 ts_tablet_manager.cc:1367] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: Time spent bootstrapping tablet: real 0.408s	user 0.148s	sys 0.021s
I20240501 20:09:46.527794 14041 raft_consensus.cc:348] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:46.528452 14041 raft_consensus.cc:729] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 535367472879403fa999dd00a155f042, State: Initialized, Role: LEARNER
I20240501 20:09:46.528753 14041 consensus_queue.cc:260] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 1.103, Last appended by leader: 103, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:46.529372 14041 ts_tablet_manager.cc:1398] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:09:46.530117 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 535367472879403fa999dd00a155f042-67043e58e5fd4233bc945432eeae1227 received from {username='slave'} at 127.8.68.197:35537
I20240501 20:09:46.530236 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 535367472879403fa999dd00a155f042-67043e58e5fd4233bc945432eeae1227 on tablet 67043e58e5fd4233bc945432eeae1227 with peer 535367472879403fa999dd00a155f042
I20240501 20:09:46.545728 13406 raft_consensus.cc:1261] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240501 20:09:46.547999 13950 consensus_queue.cc:1035] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240501 20:09:46.553107 13782 raft_consensus.cc:2932] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.130: config changed from index 123 to 130, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:46.556568 11818 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf6c676aa51e44a9b8a1c2a3c828ccd0 with cas_config_opid_index 123: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:46.559000 11827 catalog_manager.cc:5484] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 123 to 130, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:46.560073 13407 raft_consensus.cc:2932] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 123 to 130, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:46.562733 12212 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6c676aa51e44a9b8a1c2a3c828ccd0 with delete_type TABLET_DATA_TOMBSTONED (TS 18717c9754d04918815985cf07c1ddad not found in new config with opid_index 130) from {username='slave'} at 127.0.0.1:50996
I20240501 20:09:46.603855 12103 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:09:46.605115 11909 consensus_peers.cc:485] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 -> Peer 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351): Couldn't send request to peer 3b95f8c6589e473bb21db7681e227638. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 262853fe9d5c4aa7acdac809a6378567. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:46.606786 14101 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:09:46.644990 13916 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:46.655390 12103 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:46.657428 11827 catalog_manager.cc:5484] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:46.661957 14139 tablet_replica.cc:330] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad: stopping tablet replica
I20240501 20:09:46.662149 14139 raft_consensus.cc:2229] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:46.663012 14139 raft_consensus.cc:2258] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:46.682585 14144 ts_tablet_manager.cc:902] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: Initiating tablet copy from peer dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:46.691195 14139 ts_tablet_manager.cc:1875] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:46.717195 14144 tablet_copy_client.cc:308] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:39397
I20240501 20:09:46.731593 14139 ts_tablet_manager.cc:1888] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.129
I20240501 20:09:46.731580 11993 tablet_copy_service.cc:140] P dc71b180f72b4c4697d28b13b363d705: Received BeginTabletCopySession request for tablet 2c0a1865e31241569216a3b4b072829c from peer aed522b44a8e420daf65718450541c3c ({username='slave'} at 127.8.68.199:47273)
I20240501 20:09:46.735569 14139 log.cc:1196] T cf6c676aa51e44a9b8a1c2a3c828ccd0 P 18717c9754d04918815985cf07c1ddad: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/cf6c676aa51e44a9b8a1c2a3c828ccd0
I20240501 20:09:46.736666 11816 catalog_manager.cc:4830] TS 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201): tablet cf6c676aa51e44a9b8a1c2a3c828ccd0 (table rebalance_test_table_0 [id=801bc83cb7614cdaaf85134efd64f1d7]) successfully deleted
I20240501 20:09:46.739396 11993 tablet_copy_service.cc:161] P dc71b180f72b4c4697d28b13b363d705: Beginning new tablet copy session on tablet 2c0a1865e31241569216a3b4b072829c from peer aed522b44a8e420daf65718450541c3c at {username='slave'} at 127.8.68.199:47273: session id = aed522b44a8e420daf65718450541c3c-2c0a1865e31241569216a3b4b072829c
I20240501 20:09:46.749614 11993 tablet_copy_source_session.cc:205] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:46.754771 14144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0a1865e31241569216a3b4b072829c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.765736 14144 tablet_copy_client.cc:783] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:46.766167 14144 tablet_copy_client.cc:647] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:46.773080 14144 tablet_copy_client.cc:523] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:46.777845 14144 tablet_bootstrap.cc:492] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: Bootstrap starting.
I20240501 20:09:46.806895 14095 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=102 overwritten=0 applied=102 ignored=0} inserts{seen=787 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:46.812031 14095 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: Bootstrap complete.
I20240501 20:09:46.812314 14095 ts_tablet_manager.cc:1367] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: Time spent bootstrapping tablet: real 0.874s	user 0.174s	sys 0.014s
I20240501 20:09:46.839478 13276 raft_consensus.cc:1203] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.70->[1.71-1.94]   Dedup: 1.79->[1.80-1.94]
I20240501 20:09:46.880720 13407 raft_consensus.cc:1203] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.99->[1.100-1.122]   Dedup: 1.103->[1.104-1.122]
I20240501 20:09:46.863356 14095 raft_consensus.cc:348] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:46.894383 14095 raft_consensus.cc:729] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aed522b44a8e420daf65718450541c3c, State: Initialized, Role: LEARNER
I20240501 20:09:46.894618 14095 consensus_queue.cc:260] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 1.102, Last appended by leader: 102, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:46.964423 13950 raft_consensus.cc:1050] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 535367472879403fa999dd00a155f042 to VOTER
I20240501 20:09:46.965109 13950 consensus_queue.cc:237] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:46.968662 13960 ts_tablet_manager.cc:902] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: Initiating tablet copy from peer dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:46.969900 13960 tablet_copy_client.cc:308] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:39397
I20240501 20:09:46.970124 13407 raft_consensus.cc:1261] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240501 20:09:46.979939 13754 consensus_queue.cc:1035] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240501 20:09:46.986876 14092 raft_consensus.cc:2932] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index 100 to 123, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:46.992821 11827 catalog_manager.cc:5484] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 100 to 123, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:46.996467 11993 tablet_copy_service.cc:140] P dc71b180f72b4c4697d28b13b363d705: Received BeginTabletCopySession request for tablet 262853fe9d5c4aa7acdac809a6378567 from peer 3b95f8c6589e473bb21db7681e227638 ({username='slave'} at 127.8.68.196:40913)
I20240501 20:09:46.996654 11993 tablet_copy_service.cc:161] P dc71b180f72b4c4697d28b13b363d705: Beginning new tablet copy session on tablet 262853fe9d5c4aa7acdac809a6378567 from peer 3b95f8c6589e473bb21db7681e227638 at {username='slave'} at 127.8.68.196:40913: session id = 3b95f8c6589e473bb21db7681e227638-262853fe9d5c4aa7acdac809a6378567
I20240501 20:09:46.999617 11993 tablet_copy_source_session.cc:205] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:47.001179 11973 raft_consensus.cc:1261] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240501 20:09:47.004323 13960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262853fe9d5c4aa7acdac809a6378567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.025259 14078 raft_consensus.cc:1050] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 3b95f8c6589e473bb21db7681e227638 to VOTER
I20240501 20:09:47.025931 14078 consensus_queue.cc:237] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:47.029249 13277 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240501 20:09:47.029743 14092 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 20:09:47.028667 13960 tablet_copy_client.cc:783] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:47.030241 13960 tablet_copy_client.cc:647] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:47.032137 13782 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 71 to 97, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:47.033262 13277 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 71 to 97, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:47.034929 13960 tablet_copy_client.cc:523] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:47.039093 11828 catalog_manager.cc:5484] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 71 to 97, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:47.044704 12036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.197:43403, user_credentials={real_user=slave}} blocked reactor thread for 58860.9us
I20240501 20:09:47.047348 11973 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240501 20:09:47.049348 12103 consensus_queue.cc:237] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:47.053260 13406 raft_consensus.cc:2932] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 100 to 123, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:47.053869 14095 ts_tablet_manager.cc:1398] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c: Time spent starting tablet: real 0.241s	user 0.006s	sys 0.000s
I20240501 20:09:47.054905 11993 tablet_copy_service.cc:342] P dc71b180f72b4c4697d28b13b363d705: Request end of tablet copy session aed522b44a8e420daf65718450541c3c-f8f9c24067b442299db6fa5831feb56d received from {username='slave'} at 127.8.68.199:47273
I20240501 20:09:47.055068 11993 tablet_copy_service.cc:433] P dc71b180f72b4c4697d28b13b363d705: ending tablet copy session aed522b44a8e420daf65718450541c3c-f8f9c24067b442299db6fa5831feb56d on tablet f8f9c24067b442299db6fa5831feb56d with peer aed522b44a8e420daf65718450541c3c
I20240501 20:09:47.058404 13277 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 20:09:47.059383 13754 consensus_queue.cc:1035] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240501 20:09:47.059581 14092 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240501 20:09:47.060882 13960 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: Bootstrap starting.
I20240501 20:09:47.061702 14092 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index 97 to 98, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:47.066882 11818 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7eb828d8511f40c78004d0aaba843df5 with cas_config_opid_index 97: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:47.067087 13276 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 97 to 98, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:47.068903 11827 catalog_manager.cc:5484] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 97 to 98, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:47.073513 11952 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb828d8511f40c78004d0aaba843df5 with delete_type TABLET_DATA_TOMBSTONED (TS dc71b180f72b4c4697d28b13b363d705 not found in new config with opid_index 98) from {username='slave'} at 127.0.0.1:58526
I20240501 20:09:47.081815 11972 raft_consensus.cc:2932] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 100 to 123, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:47.091042 14178 tablet_replica.cc:330] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: stopping tablet replica
I20240501 20:09:47.091984 14178 raft_consensus.cc:2229] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.107628 14178 pending_rounds.cc:62] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: Trying to abort 1 pending ops.
I20240501 20:09:47.107704 14178 pending_rounds.cc:68] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: Aborting op as it isn't in flight: id { term: 1 index: 96 } timestamp: 7022977789798764544 op_type: WRITE_OP write_request { tablet_id: "7eb828d8511f40c78004d0aaba843df5" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000+\340h\t,\242\2633\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5b374fe998bef190" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977789725093888 authz_token { token_data: "\010\260\305\312\261\006\"3\n\005slave\022*\n 0e909cfb82614206afe575f03aece87c\020\001\030\001 \001(\001" signature: "<redacted>""\027\362\215*\272\343\302\325~d\352f\363\307D\3075\2469~\245c:\207\016\036&\273a\032\356\250\034\363U\265\244\3264\365\2564 \230D\222\204\232z\334\373\277I\244\315\265|\205\366&\254\\x\334" signing_key_seq_num: 0 } } request_id { client_id: "ffa1738c07b84921a69b2af74b85df88" seq_no: 314 first_incomplete_seq_no: 314 attempt_no: 0 }
I20240501 20:09:47.108220 14178 raft_consensus.cc:2258] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.109414 14178 ts_tablet_manager.cc:1875] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:47.132427 14178 ts_tablet_manager.cc:1888] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.96
I20240501 20:09:47.132618 14178 log.cc:1196] T 7eb828d8511f40c78004d0aaba843df5 P dc71b180f72b4c4697d28b13b363d705: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/7eb828d8511f40c78004d0aaba843df5
I20240501 20:09:47.133777 11816 catalog_manager.cc:4830] TS dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397): tablet 7eb828d8511f40c78004d0aaba843df5 (table rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]) successfully deleted
I20240501 20:09:47.228219 14144 tablet_bootstrap.cc:492] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=136 overwritten=0 applied=136 ignored=0} inserts{seen=808 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:47.228716 14144 tablet_bootstrap.cc:492] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: Bootstrap complete.
I20240501 20:09:47.229015 14144 ts_tablet_manager.cc:1367] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: Time spent bootstrapping tablet: real 0.451s	user 0.183s	sys 0.014s
I20240501 20:09:47.229758 14144 raft_consensus.cc:348] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:47.230000 14144 raft_consensus.cc:729] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aed522b44a8e420daf65718450541c3c, State: Initialized, Role: LEARNER
I20240501 20:09:47.230185 14144 consensus_queue.cc:260] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 136, Last appended: 1.136, Last appended by leader: 136, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: true } }
I20240501 20:09:47.230734 14144 ts_tablet_manager.cc:1398] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:09:47.233094 11993 tablet_copy_service.cc:342] P dc71b180f72b4c4697d28b13b363d705: Request end of tablet copy session aed522b44a8e420daf65718450541c3c-2c0a1865e31241569216a3b4b072829c received from {username='slave'} at 127.8.68.199:47273
I20240501 20:09:47.233232 11993 tablet_copy_service.cc:433] P dc71b180f72b4c4697d28b13b363d705: ending tablet copy session aed522b44a8e420daf65718450541c3c-2c0a1865e31241569216a3b4b072829c on tablet 2c0a1865e31241569216a3b4b072829c with peer aed522b44a8e420daf65718450541c3c
I20240501 20:09:47.266633 13668 raft_consensus.cc:1203] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.83->[1.84-1.133]   Dedup: 1.102->[1.103-1.133]
I20240501 20:09:47.280267 13960 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:47.280730 13960 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: Bootstrap complete.
I20240501 20:09:47.280978 13960 ts_tablet_manager.cc:1367] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: Time spent bootstrapping tablet: real 0.220s	user 0.023s	sys 0.041s
I20240501 20:09:47.281726 13960 raft_consensus.cc:348] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:47.281977 13960 raft_consensus.cc:729] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b95f8c6589e473bb21db7681e227638, State: Initialized, Role: LEARNER
I20240501 20:09:47.282204 13960 consensus_queue.cc:260] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:47.282850 13960 ts_tablet_manager.cc:1398] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240501 20:09:47.283967 11993 tablet_copy_service.cc:342] P dc71b180f72b4c4697d28b13b363d705: Request end of tablet copy session 3b95f8c6589e473bb21db7681e227638-262853fe9d5c4aa7acdac809a6378567 received from {username='slave'} at 127.8.68.196:40913
I20240501 20:09:47.284103 11993 tablet_copy_service.cc:433] P dc71b180f72b4c4697d28b13b363d705: ending tablet copy session 3b95f8c6589e473bb21db7681e227638-262853fe9d5c4aa7acdac809a6378567 on tablet 262853fe9d5c4aa7acdac809a6378567 with peer 3b95f8c6589e473bb21db7681e227638
I20240501 20:09:47.359622 13916 raft_consensus.cc:1050] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: attempting to promote NON_VOTER aed522b44a8e420daf65718450541c3c to VOTER
I20240501 20:09:47.360483 13916 consensus_queue.cc:237] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 137, Committed index: 137, Last appended: 1.137, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:47.362607 12232 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:47.363160 14167 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:47.363624 13667 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:47.365427 14167 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:47.367868 14167 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.138: config changed from index 84 to 138, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.372045 13668 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index 84 to 138, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.376722 12231 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index 84 to 138, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.377290 11827 catalog_manager.cc:5484] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c reported cstate change: config changed from index 84 to 138, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.387073 11969 consensus_queue.cc:237] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 138, Committed index: 138, Last appended: 1.138, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:47.389125 13668 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240501 20:09:47.390481 14051 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240501 20:09:47.402208 14101 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.139: config changed from index 138 to 139, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.403512 13276 raft_consensus.cc:1203] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 20:09:47.406884 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8f9c24067b442299db6fa5831feb56d with cas_config_opid_index 138: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:47.409178 11827 catalog_manager.cc:5484] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index 138 to 139, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:47.410634 13668 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 138 to 139, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.417219 12212 tablet_service.cc:1514] Processing DeleteTablet for tablet f8f9c24067b442299db6fa5831feb56d with delete_type TABLET_DATA_TOMBSTONED (TS 18717c9754d04918815985cf07c1ddad not found in new config with opid_index 139) from {username='slave'} at 127.0.0.1:50996
I20240501 20:09:47.456261 14195 tablet_replica.cc:330] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad: stopping tablet replica
I20240501 20:09:47.456455 14195 raft_consensus.cc:2229] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.456846 14195 raft_consensus.cc:2258] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.457870 14195 ts_tablet_manager.cc:1875] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:47.470619 14195 ts_tablet_manager.cc:1888] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.138
I20240501 20:09:47.470814 14195 log.cc:1196] T f8f9c24067b442299db6fa5831feb56d P 18717c9754d04918815985cf07c1ddad: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/f8f9c24067b442299db6fa5831feb56d
I20240501 20:09:47.472440 11816 catalog_manager.cc:4830] TS 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201): tablet f8f9c24067b442299db6fa5831feb56d (table rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]) successfully deleted
I20240501 20:09:47.569804 12103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56498
I20240501 20:09:47.570067 12103 raft_consensus.cc:595] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Received request to transfer leadership
W20240501 20:09:47.582144 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56328: Service unavailable: leader transfer in progress
W20240501 20:09:47.585387 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56328: Service unavailable: leader transfer in progress
I20240501 20:09:47.586124 14189 raft_consensus.cc:979] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: : Instructing follower dc71b180f72b4c4697d28b13b363d705 to start an election
I20240501 20:09:47.586264 14189 raft_consensus.cc:1067] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Signalling peer dc71b180f72b4c4697d28b13b363d705 to start an election
W20240501 20:09:47.590698 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56328: Service unavailable: leader transfer in progress
I20240501 20:09:47.598999 11973 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e"
dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
 from {username='slave'} at 127.8.68.194:34101
I20240501 20:09:47.599261 11973 raft_consensus.cc:482] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:47.599387 11973 raft_consensus.cc:3037] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:09:47.599498 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56328: Service unavailable: leader transfer in progress
I20240501 20:09:47.602142 11973 raft_consensus.cc:504] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:47.603206 11973 leader_election.cc:288] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 2 election: Requested vote from peers 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853), 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351)
I20240501 20:09:47.605796 13276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 2 candidate_status { last_received { term: 1 index: 110 } } ignore_live_leader: true dest_uuid: "3b95f8c6589e473bb21db7681e227638"
I20240501 20:09:47.606191 13276 raft_consensus.cc:3037] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:09:47.609299 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56328: Service unavailable: leader transfer in progress
I20240501 20:09:47.613006 13276 raft_consensus.cc:2454] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 2.
I20240501 20:09:47.613968 11909 leader_election.cc:302] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b95f8c6589e473bb21db7681e227638, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:47.614984 12103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 2 candidate_status { last_received { term: 1 index: 110 } } ignore_live_leader: true dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
I20240501 20:09:47.615185 12103 raft_consensus.cc:3032] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:47.615253 12103 raft_consensus.cc:729] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:47.615553 12103 consensus_queue.cc:260] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:47.615830 12103 raft_consensus.cc:3037] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:47.618113 12103 raft_consensus.cc:2454] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 2.
I20240501 20:09:47.619853 14051 raft_consensus.cc:2781] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:47.620018 14051 raft_consensus.cc:686] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Becoming Leader. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Running, Role: LEADER
I20240501 20:09:47.620270 14051 consensus_queue.cc:237] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:47.625092 11827 catalog_manager.cc:5484] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 reported cstate change: term changed from 1 to 2, leader changed from 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) to dc71b180f72b4c4697d28b13b363d705 (127.8.68.193). New cstate: current_term: 2 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:47.630695 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56328: Illegal state: replica 34f8af7f2d4245899a7d8fd5012a3721 is not leader of this config: current role FOLLOWER
I20240501 20:09:47.646243 13667 raft_consensus.cc:1203] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.128->[1.129-1.157]   Dedup: 1.136->[1.137-1.157]
I20240501 20:09:47.715767 13276 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 2 index: 112. (index mismatch)
I20240501 20:09:47.716641 12103 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 2 index: 112. (index mismatch)
I20240501 20:09:47.718072 14167 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240501 20:09:47.719614 14167 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240501 20:09:47.724771 14023 mvcc.cc:205] Tried to move back new op lower bound from 7022977792879026176 to 7022977792494993408. Current Snapshot: MvccSnapshot[applied={T|T < 7022977792879026176 or (T in {7022977792879026176})}]
I20240501 20:09:47.725611 13809 mvcc.cc:205] Tried to move back new op lower bound from 7022977792879026176 to 7022977792494993408. Current Snapshot: MvccSnapshot[applied={T|T < 7022977792879026176 or (T in {7022977792879026176})}]
I20240501 20:09:47.726608 11973 consensus_queue.cc:237] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 2.112, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:47.731474 13276 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 2 index: 112. Preceding OpId from leader: term: 2 index: 113. (index mismatch)
I20240501 20:09:47.732625 14051 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20240501 20:09:47.739791 13916 raft_consensus.cc:1050] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705: attempting to promote NON_VOTER aed522b44a8e420daf65718450541c3c to VOTER
I20240501 20:09:47.740187 13916 consensus_queue.cc:237] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.158, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:47.742202 14101 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Committing config change with OpId 2.113: config changed from index 80 to 113, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:47.744519 13277 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Committing config change with OpId 2.113: config changed from index 80 to 113, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:47.746920 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d7dd13822784e789d558b4cc6c5010e with cas_config_opid_index 80: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:47.747879 12232 raft_consensus.cc:1261] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240501 20:09:47.749204 13668 raft_consensus.cc:1261] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c [term 1 LEARNER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240501 20:09:47.751534 14051 consensus_queue.cc:1035] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I20240501 20:09:47.751948 14051 consensus_queue.cc:1035] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I20240501 20:09:47.757261 14051 raft_consensus.cc:2932] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.159: config changed from index 129 to 159, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.759660 12232 raft_consensus.cc:2932] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 129 to 159, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.761801 11827 catalog_manager.cc:5484] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 reported cstate change: config changed from index 80 to 113, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:47.778074 13668 raft_consensus.cc:2932] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 129 to 159, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.771504 11829 catalog_manager.cc:5484] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index 129 to 159, aed522b44a8e420daf65718450541c3c (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.802309 12083 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7dd13822784e789d558b4cc6c5010e with delete_type TABLET_DATA_TOMBSTONED (TS 34f8af7f2d4245899a7d8fd5012a3721 not found in new config with opid_index 113) from {username='slave'} at 127.0.0.1:42804
I20240501 20:09:47.807996 11969 consensus_queue.cc:237] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 1.160, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:47.812451 13667 raft_consensus.cc:1261] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240501 20:09:47.817605 14101 consensus_queue.cc:1035] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240501 20:09:47.824509 14225 tablet_replica.cc:330] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: stopping tablet replica
I20240501 20:09:47.827464 14167 raft_consensus.cc:2932] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.161: config changed from index 159 to 161, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.829614 14225 raft_consensus.cc:2229] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.829943 14225 pending_rounds.cc:62] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Trying to abort 2 pending ops.
I20240501 20:09:47.830109 14225 pending_rounds.cc:68] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Aborting op as it isn't in flight: id { term: 2 index: 112 } timestamp: 7022977792879026176 op_type: WRITE_OP write_request { tablet_id: "9d7dd13822784e789d558b4cc6c5010e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\357\233\272A\275Gp\372\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""0606a7ff3dfeb2e0" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977792231002112 authz_token { token_data: "\010\260\305\312\261\006\"3\n\005slave\022*\n ba05ecbd1c2c41e293b956c75a0f3457\020\001\030\001 \001(\001" signature: "<redacted>""\267?\347\217\360]bHZ\263\327,\242;%#GbH\213\375\334\221\206\260 \227\220\236y\003\352!\206\273n\320\034$\257\270\2310\026\253\316\254\276Sk!\030\020epRo9\201\326N\001d\271" signing_key_seq_num: 0 } } request_id { client_id: "fa122e7e00d749a8a06840f2d7f114ac" seq_no: 292 first_incomplete_seq_no: 292 attempt_no: 6 }
I20240501 20:09:47.831123 14225 pending_rounds.cc:68] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Aborting op as it isn't in flight: id { term: 2 index: 111 } timestamp: 7022977792494993408 op_type: NO_OP noop_request { }
I20240501 20:09:47.831312 14225 raft_consensus.cc:2866] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:09:47.831488 14225 raft_consensus.cc:2258] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.832286 13668 raft_consensus.cc:2932] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 159 to 161, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:47.835498 14225 ts_tablet_manager.cc:1875] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:47.836398 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c0a1865e31241569216a3b4b072829c with cas_config_opid_index 159: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:47.839287 11828 catalog_manager.cc:5484] T 2c0a1865e31241569216a3b4b072829c P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index 159 to 161, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:47.851142 14225 ts_tablet_manager.cc:1888] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.112
I20240501 20:09:47.851290 14225 log.cc:1196] T 9d7dd13822784e789d558b4cc6c5010e P 34f8af7f2d4245899a7d8fd5012a3721: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/9d7dd13822784e789d558b4cc6c5010e
I20240501 20:09:47.852674 12212 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c0a1865e31241569216a3b4b072829c with delete_type TABLET_DATA_TOMBSTONED (TS 18717c9754d04918815985cf07c1ddad not found in new config with opid_index 161) from {username='slave'} at 127.0.0.1:50996
I20240501 20:09:47.852810 11818 catalog_manager.cc:4830] TS 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853): tablet 9d7dd13822784e789d558b4cc6c5010e (table rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]) successfully deleted
I20240501 20:09:47.853339 14195 tablet_replica.cc:330] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: stopping tablet replica
I20240501 20:09:47.853528 14195 raft_consensus.cc:2229] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.853781 14195 raft_consensus.cc:2258] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.856228 14195 ts_tablet_manager.cc:1875] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:47.899767 14195 ts_tablet_manager.cc:1888] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.160
I20240501 20:09:47.899995 14195 log.cc:1196] T 2c0a1865e31241569216a3b4b072829c P 18717c9754d04918815985cf07c1ddad: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/2c0a1865e31241569216a3b4b072829c
I20240501 20:09:47.900820 11816 catalog_manager.cc:4830] TS 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201): tablet 2c0a1865e31241569216a3b4b072829c (table rebalance_test_table_0 [id=801bc83cb7614cdaaf85134efd64f1d7]) successfully deleted
I20240501 20:09:48.016371 14051 raft_consensus.cc:1050] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: attempting to promote NON_VOTER 3b95f8c6589e473bb21db7681e227638 to VOTER
I20240501 20:09:48.017148 14051 consensus_queue.cc:237] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:48.039188 13277 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:48.039676 14101 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:48.042793 12103 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:09:48.045132 13916 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:09:48.064095 14051 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:48.070742 12103 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:48.075932 11827 catalog_manager.cc:5484] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index 2 to 3, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:48.143817 13275 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:48.315371 12103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38b8e2865d134700ad0959372afc6670"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56528
I20240501 20:09:48.315774 12103 raft_consensus.cc:595] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Received request to transfer leadership
W20240501 20:09:48.334250 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.351755 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.367586 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.386191 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.403184 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.440078 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
I20240501 20:09:48.442013 12103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67043e58e5fd4233bc945432eeae1227"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56548
I20240501 20:09:48.442212 12103 raft_consensus.cc:595] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Received request to transfer leadership
W20240501 20:09:48.453344 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.466495 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.483485 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.494619 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.510517 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.525462 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.542433 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.559603 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.579068 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.600894 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.620801 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.640627 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.665668 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.698101 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.725629 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
I20240501 20:09:48.741909 12103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "575f6e5055d84dc38ef2326cec06fa47"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56578
I20240501 20:09:48.742133 12103 raft_consensus.cc:595] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:48.743333 14189 raft_consensus.cc:979] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: : Instructing follower dc71b180f72b4c4697d28b13b363d705 to start an election
I20240501 20:09:48.743465 14189 raft_consensus.cc:1067] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Signalling peer dc71b180f72b4c4697d28b13b363d705 to start an election
I20240501 20:09:48.745048 11969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "575f6e5055d84dc38ef2326cec06fa47"
dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
 from {username='slave'} at 127.8.68.194:34101
I20240501 20:09:48.745381 11969 raft_consensus.cc:482] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:48.745538 11969 raft_consensus.cc:3037] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:09:48.747462 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56328: Service unavailable: leader transfer in progress
I20240501 20:09:48.748812 11969 raft_consensus.cc:504] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
W20240501 20:09:48.748896 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
I20240501 20:09:48.749528 11969 leader_election.cc:288] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 2 election: Requested vote from peers 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853), aed522b44a8e420daf65718450541c3c (127.8.68.199:40105)
I20240501 20:09:48.750416 12103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575f6e5055d84dc38ef2326cec06fa47" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 2 candidate_status { last_received { term: 1 index: 120 } } ignore_live_leader: true dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
I20240501 20:09:48.750612 12103 raft_consensus.cc:3032] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:48.750720 12103 raft_consensus.cc:729] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:48.750980 12103 consensus_queue.cc:260] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.120, Last appended by leader: 120, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:48.751322 12103 raft_consensus.cc:3037] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.753361 12103 raft_consensus.cc:2454] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 2.
I20240501 20:09:48.753780 11908 leader_election.cc:302] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:48.754076 13916 raft_consensus.cc:2781] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:48.754194 13916 raft_consensus.cc:686] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Becoming Leader. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Running, Role: LEADER
I20240501 20:09:48.754441 13916 consensus_queue.cc:237] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.120, Last appended by leader: 120, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:48.760084 11828 catalog_manager.cc:5484] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 reported cstate change: term changed from 1 to 2, leader changed from 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) to dc71b180f72b4c4697d28b13b363d705 (127.8.68.193). New cstate: current_term: 2 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:48.761569 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56328: Illegal state: replica 34f8af7f2d4245899a7d8fd5012a3721 is not leader of this config: current role FOLLOWER
I20240501 20:09:48.762038 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575f6e5055d84dc38ef2326cec06fa47" candidate_uuid: "dc71b180f72b4c4697d28b13b363d705" candidate_term: 2 candidate_status { last_received { term: 1 index: 120 } } ignore_live_leader: true dest_uuid: "aed522b44a8e420daf65718450541c3c"
I20240501 20:09:48.762393 13668 raft_consensus.cc:3037] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.767166 13668 raft_consensus.cc:2454] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc71b180f72b4c4697d28b13b363d705 in term 2.
I20240501 20:09:48.768611 12103 raft_consensus.cc:1261] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 2 index: 122. (index mismatch)
I20240501 20:09:48.769184 13667 raft_consensus.cc:1261] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 2 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 2 index: 122. (index mismatch)
I20240501 20:09:48.769330 13916 consensus_queue.cc:1035] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20240501 20:09:48.770195 13916 consensus_queue.cc:1035] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
W20240501 20:09:48.775846 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
I20240501 20:09:48.788926 14187 mvcc.cc:205] Tried to move back new op lower bound from 7022977797189591040 to 7022977797140594688. Current Snapshot: MvccSnapshot[applied={T|T < 7022977797189591040 or (T in {7022977797189591040})}]
I20240501 20:09:48.789670 13818 raft_consensus.cc:979] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: : Instructing follower 535367472879403fa999dd00a155f042 to start an election
I20240501 20:09:48.789824 14189 raft_consensus.cc:1067] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Signalling peer 535367472879403fa999dd00a155f042 to start an election
I20240501 20:09:48.790599 13406 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67043e58e5fd4233bc945432eeae1227"
dest_uuid: "535367472879403fa999dd00a155f042"
 from {username='slave'} at 127.8.68.194:42397
I20240501 20:09:48.790858 13406 raft_consensus.cc:482] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:48.791067 13406 raft_consensus.cc:3037] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.797792 13406 raft_consensus.cc:504] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:48.800928 12102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67043e58e5fd4233bc945432eeae1227" candidate_uuid: "535367472879403fa999dd00a155f042" candidate_term: 2 candidate_status { last_received { term: 1 index: 142 } } ignore_live_leader: true dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
I20240501 20:09:48.801169 12102 raft_consensus.cc:3032] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:48.801268 12102 raft_consensus.cc:729] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:48.801546 12102 consensus_queue.cc:260] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 1.142, Last appended by leader: 142, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:48.801899 12102 raft_consensus.cc:3037] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.804767 12102 raft_consensus.cc:2454] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 535367472879403fa999dd00a155f042 in term 2.
I20240501 20:09:48.806018 13343 leader_election.cc:302] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, 535367472879403fa999dd00a155f042; no voters: 
I20240501 20:09:48.806865 14279 raft_consensus.cc:2781] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 20:09:48.807297 12061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
I20240501 20:09:48.813370 13406 leader_election.cc:288] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [CANDIDATE]: Term 2 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397), 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:48.815160 14279 raft_consensus.cc:686] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 2 LEADER]: Becoming Leader. State: Replica: 535367472879403fa999dd00a155f042, State: Running, Role: LEADER
I20240501 20:09:48.815724 14279 consensus_queue.cc:237] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 1.142, Last appended by leader: 142, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:48.820209 11829 catalog_manager.cc:5484] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 reported cstate change: term changed from 1 to 2, leader changed from 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) to 535367472879403fa999dd00a155f042 (127.8.68.197). New cstate: current_term: 2 leader_uuid: "535367472879403fa999dd00a155f042" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:48.834401 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
W20240501 20:09:48.862650 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
I20240501 20:09:48.877224 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67043e58e5fd4233bc945432eeae1227" candidate_uuid: "535367472879403fa999dd00a155f042" candidate_term: 2 candidate_status { last_received { term: 1 index: 142 } } ignore_live_leader: true dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:48.877430 11969 raft_consensus.cc:3037] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.880005 11969 raft_consensus.cc:2454] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 535367472879403fa999dd00a155f042 in term 2.
I20240501 20:09:48.881310 11969 consensus_queue.cc:237] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 2.124, Last appended by leader: 120, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } }
I20240501 20:09:48.885823 13667 raft_consensus.cc:1261] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 2 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 2 index: 124. Preceding OpId from leader: term: 2 index: 125. (index mismatch)
I20240501 20:09:48.888464 14051 consensus_queue.cc:1035] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
W20240501 20:09:48.895803 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Service unavailable: leader transfer in progress
I20240501 20:09:48.898633 14051 raft_consensus.cc:2932] T 575f6e5055d84dc38ef2326cec06fa47 P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Committing config change with OpId 2.125: config changed from index 70 to 125, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:48.901146 13669 raft_consensus.cc:2932] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c [term 2 FOLLOWER]: Committing config change with OpId 2.125: config changed from index 70 to 125, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:48.907163 13818 raft_consensus.cc:979] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: : Instructing follower 463bd46a1f6e4e89a3040c4cf50dd919 to start an election
I20240501 20:09:48.906944 11828 catalog_manager.cc:5484] T 575f6e5055d84dc38ef2326cec06fa47 P aed522b44a8e420daf65718450541c3c reported cstate change: config changed from index 70 to 125, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } }
I20240501 20:09:48.907321 13818 raft_consensus.cc:1067] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Signalling peer 463bd46a1f6e4e89a3040c4cf50dd919 to start an election
I20240501 20:09:48.908277 13537 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38b8e2865d134700ad0959372afc6670"
dest_uuid: "463bd46a1f6e4e89a3040c4cf50dd919"
 from {username='slave'} at 127.8.68.194:46053
I20240501 20:09:48.908532 13537 raft_consensus.cc:482] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:48.908706 13537 raft_consensus.cc:3037] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.911294 12083 tablet_service.cc:1514] Processing DeleteTablet for tablet 575f6e5055d84dc38ef2326cec06fa47 with delete_type TABLET_DATA_TOMBSTONED (TS 34f8af7f2d4245899a7d8fd5012a3721 not found in new config with opid_index 125) from {username='slave'} at 127.0.0.1:42804
I20240501 20:09:48.911463 13537 raft_consensus.cc:504] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:48.913161 13537 leader_election.cc:288] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [CANDIDATE]: Term 2 election: Requested vote from peers 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853), 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:48.914054 12102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b8e2865d134700ad0959372afc6670" candidate_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" candidate_term: 2 candidate_status { last_received { term: 1 index: 135 } } ignore_live_leader: true dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
I20240501 20:09:48.914256 12102 raft_consensus.cc:3032] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:48.914388 12102 raft_consensus.cc:729] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:48.914510 12231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b8e2865d134700ad0959372afc6670" candidate_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" candidate_term: 2 candidate_status { last_received { term: 1 index: 135 } } ignore_live_leader: true dest_uuid: "18717c9754d04918815985cf07c1ddad"
I20240501 20:09:48.914716 12231 raft_consensus.cc:3037] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.914656 12102 consensus_queue.cc:260] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 135, Committed index: 135, Last appended: 1.135, Last appended by leader: 135, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:48.915000 12102 raft_consensus.cc:3037] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.917269 12231 raft_consensus.cc:2454] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 463bd46a1f6e4e89a3040c4cf50dd919 in term 2.
I20240501 20:09:48.917517 14289 tablet_replica.cc:330] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: stopping tablet replica
I20240501 20:09:48.917723 14289 raft_consensus.cc:2229] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:48.917848 12102 raft_consensus.cc:2454] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 463bd46a1f6e4e89a3040c4cf50dd919 in term 2.
I20240501 20:09:48.917987 14289 raft_consensus.cc:2258] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:48.918165 13471 leader_election.cc:302] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 463bd46a1f6e4e89a3040c4cf50dd919; no voters: 
I20240501 20:09:48.919071 12231 raft_consensus.cc:1226] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 2 FOLLOWER]: Rejecting Update request from peer 34f8af7f2d4245899a7d8fd5012a3721 for earlier term 1. Current term is 2. Ops: []
I20240501 20:09:48.919502 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 575f6e5055d84dc38ef2326cec06fa47 with cas_config_opid_index 70: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:48.920470 14291 raft_consensus.cc:2781] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 20:09:48.929224 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica 34f8af7f2d4245899a7d8fd5012a3721 is not leader of this config: current role FOLLOWER
I20240501 20:09:48.942720 14291 raft_consensus.cc:686] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 LEADER]: Becoming Leader. State: Replica: 463bd46a1f6e4e89a3040c4cf50dd919, State: Running, Role: LEADER
I20240501 20:09:48.943323 14291 consensus_queue.cc:237] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 135, Committed index: 135, Last appended: 1.135, Last appended by leader: 135, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:48.949558 11828 catalog_manager.cc:5484] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 reported cstate change: term changed from 1 to 2, leader changed from 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) to 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198). New cstate: current_term: 2 leader_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:48.961831 14289 ts_tablet_manager.cc:1875] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:49.021035 11973 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "262853fe9d5c4aa7acdac809a6378567"
dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40224
I20240501 20:09:49.021296 11973 raft_consensus.cc:595] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:49.037997 14289 ts_tablet_manager.cc:1888] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.124
I20240501 20:09:49.038116 14289 log.cc:1196] T 575f6e5055d84dc38ef2326cec06fa47 P 34f8af7f2d4245899a7d8fd5012a3721: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/575f6e5055d84dc38ef2326cec06fa47
I20240501 20:09:49.045567 11818 catalog_manager.cc:4830] TS 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853): tablet 575f6e5055d84dc38ef2326cec06fa47 (table rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]) successfully deleted
W20240501 20:09:49.088546 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38746: Illegal state: replica 18717c9754d04918815985cf07c1ddad is not leader of this config: current role FOLLOWER
I20240501 20:09:49.097970 13916 raft_consensus.cc:979] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: : Instructing follower 34f8af7f2d4245899a7d8fd5012a3721 to start an election
I20240501 20:09:49.098152 13916 raft_consensus.cc:1067] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Signalling peer 34f8af7f2d4245899a7d8fd5012a3721 to start an election
I20240501 20:09:49.099202 12102 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "262853fe9d5c4aa7acdac809a6378567"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
 from {username='slave'} at 127.8.68.193:51533
I20240501 20:09:49.099417 12102 raft_consensus.cc:482] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:49.099553 12102 raft_consensus.cc:3037] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.102357 12102 raft_consensus.cc:504] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:49.104131 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262853fe9d5c4aa7acdac809a6378567" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:49.104346 11969 raft_consensus.cc:3032] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:49.104427 11969 raft_consensus.cc:729] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Running, Role: LEADER
I20240501 20:09:49.104714 11969 consensus_queue.cc:260] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:49.105094 11969 raft_consensus.cc:3037] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.107226 13276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "262853fe9d5c4aa7acdac809a6378567" candidate_uuid: "34f8af7f2d4245899a7d8fd5012a3721" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b95f8c6589e473bb21db7681e227638"
I20240501 20:09:49.107420 13276 raft_consensus.cc:3037] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.109958 13276 raft_consensus.cc:2454] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 2.
I20240501 20:09:49.110495 12039 leader_election.cc:302] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, 3b95f8c6589e473bb21db7681e227638; no voters: 
I20240501 20:09:49.110853 13818 raft_consensus.cc:2781] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:49.111017 13818 raft_consensus.cc:686] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 LEADER]: Becoming Leader. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:49.111296 13818 consensus_queue.cc:237] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:49.111533 12102 leader_election.cc:288] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [CANDIDATE]: Term 2 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397), 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351)
I20240501 20:09:49.116103 11969 raft_consensus.cc:2454] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f8af7f2d4245899a7d8fd5012a3721 in term 2.
I20240501 20:09:49.135427 11828 catalog_manager.cc:5484] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: term changed from 1 to 2, leader changed from dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) to 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194). New cstate: current_term: 2 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:49.147006 12102 raft_consensus.cc:1261] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Refusing update from remote peer 463bd46a1f6e4e89a3040c4cf50dd919: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 2 index: 137. (index mismatch)
I20240501 20:09:49.147745 12231 raft_consensus.cc:1261] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 2 FOLLOWER]: Refusing update from remote peer 463bd46a1f6e4e89a3040c4cf50dd919: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 2 index: 137. (index mismatch)
I20240501 20:09:49.148391 14292 consensus_queue.cc:1035] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I20240501 20:09:49.152197 14291 consensus_queue.cc:1035] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I20240501 20:09:49.202594 13539 consensus_queue.cc:237] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 137, Committed index: 137, Last appended: 2.138, Last appended by leader: 135, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:49.210552 12232 raft_consensus.cc:1261] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 2 FOLLOWER]: Refusing update from remote peer 463bd46a1f6e4e89a3040c4cf50dd919: Log matching property violated. Preceding OpId in replica: term: 2 index: 137. Preceding OpId from leader: term: 2 index: 139. (index mismatch)
I20240501 20:09:49.222070 14291 consensus_queue.cc:1035] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:49.229885 14292 raft_consensus.cc:2932] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 LEADER]: Committing config change with OpId 2.139: config changed from index 93 to 139, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:49.234674 12232 raft_consensus.cc:2932] T 38b8e2865d134700ad0959372afc6670 P 18717c9754d04918815985cf07c1ddad [term 2 FOLLOWER]: Committing config change with OpId 2.139: config changed from index 93 to 139, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:49.236821 11819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38b8e2865d134700ad0959372afc6670 with cas_config_opid_index 93: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:49.249789 11825 catalog_manager.cc:5484] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 reported cstate change: config changed from index 93 to 139, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" committed_config { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:49.260896 12083 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b8e2865d134700ad0959372afc6670 with delete_type TABLET_DATA_TOMBSTONED (TS 34f8af7f2d4245899a7d8fd5012a3721 not found in new config with opid_index 139) from {username='slave'} at 127.0.0.1:42804
I20240501 20:09:49.261235 14289 tablet_replica.cc:330] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: stopping tablet replica
I20240501 20:09:49.261459 14289 raft_consensus.cc:2229] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:49.261708 14289 raft_consensus.cc:2258] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:49.264006 14289 ts_tablet_manager.cc:1875] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:49.274968 14289 ts_tablet_manager.cc:1888] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.137
I20240501 20:09:49.275099 14289 log.cc:1196] T 38b8e2865d134700ad0959372afc6670 P 34f8af7f2d4245899a7d8fd5012a3721: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/38b8e2865d134700ad0959372afc6670
I20240501 20:09:49.276074 11969 raft_consensus.cc:1261] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Refusing update from remote peer 535367472879403fa999dd00a155f042: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 2 index: 143. (index mismatch)
I20240501 20:09:49.276495 11818 catalog_manager.cc:4830] TS 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853): tablet 38b8e2865d134700ad0959372afc6670 (table rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]) successfully deleted
I20240501 20:09:49.277475 14279 consensus_queue.cc:1035] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I20240501 20:09:49.281154 12102 raft_consensus.cc:1261] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Refusing update from remote peer 535367472879403fa999dd00a155f042: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 2 index: 143. (index mismatch)
I20240501 20:09:49.282383 14279 consensus_queue.cc:1035] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I20240501 20:09:49.370812 13408 consensus_queue.cc:237] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 143, Committed index: 143, Last appended: 2.144, Last appended by leader: 142, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:49.374472 11973 raft_consensus.cc:1261] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Refusing update from remote peer 535367472879403fa999dd00a155f042: Log matching property violated. Preceding OpId in replica: term: 2 index: 143. Preceding OpId from leader: term: 2 index: 145. (index mismatch)
I20240501 20:09:49.375125 14280 consensus_queue.cc:1035] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 143, Time since last communication: 0.000s
I20240501 20:09:49.380363 11969 raft_consensus.cc:1203] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.143->[2.144-2.144]   Dedup: 2.144->[]
I20240501 20:09:49.382112 11969 raft_consensus.cc:2932] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Committing config change with OpId 2.145: config changed from index 123 to 145, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:49.390736 11825 catalog_manager.cc:5484] T 67043e58e5fd4233bc945432eeae1227 P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index 123 to 145, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "535367472879403fa999dd00a155f042" committed_config { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:49.390489 14332 raft_consensus.cc:2932] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 [term 2 LEADER]: Committing config change with OpId 2.145: config changed from index 123 to 145, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:49.397140 12083 tablet_service.cc:1514] Processing DeleteTablet for tablet 67043e58e5fd4233bc945432eeae1227 with delete_type TABLET_DATA_TOMBSTONED (TS 34f8af7f2d4245899a7d8fd5012a3721 not found in new config with opid_index 145) from {username='slave'} at 127.0.0.1:42804
I20240501 20:09:49.397476 14289 tablet_replica.cc:330] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: stopping tablet replica
I20240501 20:09:49.397655 14289 raft_consensus.cc:2229] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:49.397799 14289 pending_rounds.cc:62] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: Trying to abort 1 pending ops.
I20240501 20:09:49.397846 14289 pending_rounds.cc:68] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: Aborting op as it isn't in flight: id { term: 2 index: 144 } timestamp: 7022977799538454528 op_type: WRITE_OP write_request { tablet_id: "67043e58e5fd4233bc945432eeae1227" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000:FbP\262\022|O\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5da53bf896088da0" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977799231307776 authz_token { token_data: "\010\260\305\312\261\006\"3\n\005slave\022*\n 0e909cfb82614206afe575f03aece87c\020\001\030\001 \001(\001" signature: "<redacted>""\027\362\215*\272\343\302\325~d\352f\363\307D\3075\2469~\245c:\207\016\036&\273a\032\356\250\034\363U\265\244\3264\365\2564 \230D\222\204\232z\334\373\277I\244\315\265|\205\366&\254\\x\334" signing_key_seq_num: 0 } } request_id { client_id: "ffa1738c07b84921a69b2af74b85df88" seq_no: 378 first_incomplete_seq_no: 378 attempt_no: 0 }
I20240501 20:09:49.398216 14289 raft_consensus.cc:2258] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:49.401883 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67043e58e5fd4233bc945432eeae1227 with cas_config_opid_index 123: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:49.409010 14289 ts_tablet_manager.cc:1875] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:49.425949 14289 ts_tablet_manager.cc:1888] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.144
I20240501 20:09:49.426155 14289 log.cc:1196] T 67043e58e5fd4233bc945432eeae1227 P 34f8af7f2d4245899a7d8fd5012a3721: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/67043e58e5fd4233bc945432eeae1227
I20240501 20:09:49.426903 11818 catalog_manager.cc:4830] TS 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853): tablet 67043e58e5fd4233bc945432eeae1227 (table rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]) successfully deleted
I20240501 20:09:49.511713 11973 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:09:49.512346 13782 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:49.537357 13274 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:09:49.541100 14329 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:09:49.982924 12102 consensus_queue.cc:237] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:49.984874 13275 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:09:49.986063 14189 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:49.988605 14329 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:49.989419 13275 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:50.000360 11825 catalog_manager.cc:5484] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 reported cstate change: config changed from index 3 to 5, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:50.010468 11818 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 262853fe9d5c4aa7acdac809a6378567 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:50.035697 11952 tablet_service.cc:1514] Processing DeleteTablet for tablet 262853fe9d5c4aa7acdac809a6378567 with delete_type TABLET_DATA_TOMBSTONED (TS dc71b180f72b4c4697d28b13b363d705 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58526
I20240501 20:09:50.036372 14380 tablet_replica.cc:330] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: stopping tablet replica
I20240501 20:09:50.036635 14380 raft_consensus.cc:2229] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:50.056317 14380 raft_consensus.cc:2258] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:50.057397 14380 ts_tablet_manager.cc:1875] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:50.080039 14380 ts_tablet_manager.cc:1888] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 20:09:50.081774 14380 log.cc:1196] T 262853fe9d5c4aa7acdac809a6378567 P dc71b180f72b4c4697d28b13b363d705: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/262853fe9d5c4aa7acdac809a6378567
I20240501 20:09:50.083377 11816 catalog_manager.cc:4830] TS dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397): tablet 262853fe9d5c4aa7acdac809a6378567 (table TestTable [id=6791900190fd42098c6b5729056d1534]) successfully deleted
I20240501 20:09:50.621070 12212 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:50.649688 12083 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:09:50.733587 13257 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:50.794426 11952 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:50.802731 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:50.851866 13519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:50.912817 13388 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:51.058955 11973 consensus_queue.cc:237] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 185, Committed index: 185, Last appended: 2.185, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:51.085319 13274 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 2 index: 185. Preceding OpId from leader: term: 2 index: 186. (index mismatch)
I20240501 20:09:51.092733 14051 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.000s
I20240501 20:09:51.101480 14051 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Committing config change with OpId 2.186: config changed from index 113 to 186, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } } }
I20240501 20:09:51.107465 13274 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Committing config change with OpId 2.186: config changed from index 113 to 186, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } } }
I20240501 20:09:51.107964 11828 catalog_manager.cc:5484] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index 113 to 186, NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) added. New cstate: current_term: 2 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:51.135558 11909 consensus_peers.cc:485] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 -> Peer 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198:35531): Couldn't send request to peer 463bd46a1f6e4e89a3040c4cf50dd919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d7dd13822784e789d558b4cc6c5010e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:51.351617 12231 consensus_queue.cc:237] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.219, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:51.354542 13408 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 1 index: 220. (index mismatch)
I20240501 20:09:51.355113 14191 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 218, Time since last communication: 0.000s
I20240501 20:09:51.358551 14191 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.220: config changed from index 114 to 220, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:51.359010 13408 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Committing config change with OpId 1.220: config changed from index 114 to 220, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
I20240501 20:09:51.363348 11827 catalog_manager.cc:5484] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 reported cstate change: config changed from index 114 to 220, NON_VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } } }
W20240501 20:09:51.374426 12169 consensus_peers.cc:485] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad -> Peer 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351): Couldn't send request to peer 3b95f8c6589e473bb21db7681e227638. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0faf9b33582449ee9ffa353ae6ea3d6a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:51.473999 12102 consensus_queue.cc:237] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 1.221, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
W20240501 20:09:51.475982 12036 consensus_peers.cc:485] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 535367472879403fa999dd00a155f042 (127.8.68.197:43403): Couldn't send request to peer 535367472879403fa999dd00a155f042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8baf6913f38a42c3a369cd673ff9438d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:51.476158 13274 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240501 20:09:51.476725 13818 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240501 20:09:51.480755 14189 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.222: config changed from index 72 to 222, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } } }
I20240501 20:09:51.481321 13274 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 72 to 222, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } } }
I20240501 20:09:51.487265 11827 catalog_manager.cc:5484] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 72 to 222, NON_VOTER 535367472879403fa999dd00a155f042 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:51.596156 14465 ts_tablet_manager.cc:902] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: Initiating tablet copy from peer dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:09:51.651505 14465 tablet_copy_client.cc:308] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:39397
I20240501 20:09:51.683694 11993 tablet_copy_service.cc:140] P dc71b180f72b4c4697d28b13b363d705: Received BeginTabletCopySession request for tablet 9d7dd13822784e789d558b4cc6c5010e from peer 463bd46a1f6e4e89a3040c4cf50dd919 ({username='slave'} at 127.8.68.198:59931)
I20240501 20:09:51.683885 11993 tablet_copy_service.cc:161] P dc71b180f72b4c4697d28b13b363d705: Beginning new tablet copy session on tablet 9d7dd13822784e789d558b4cc6c5010e from peer 463bd46a1f6e4e89a3040c4cf50dd919 at {username='slave'} at 127.8.68.198:59931: session id = 463bd46a1f6e4e89a3040c4cf50dd919-9d7dd13822784e789d558b4cc6c5010e
I20240501 20:09:51.695268 11993 tablet_copy_source_session.cc:205] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:51.730155 14465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d7dd13822784e789d558b4cc6c5010e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.738108 14465 tablet_copy_client.cc:783] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:51.738544 14465 tablet_copy_client.cc:647] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:51.748411 14465 tablet_copy_client.cc:523] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:51.758323 14465 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap starting.
I20240501 20:09:52.012285 14494 ts_tablet_manager.cc:902] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: Initiating tablet copy from peer 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201)
I20240501 20:09:52.103751 14494 tablet_copy_client.cc:308] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:42201
I20240501 20:09:52.156003 12253 tablet_copy_service.cc:140] P 18717c9754d04918815985cf07c1ddad: Received BeginTabletCopySession request for tablet 0faf9b33582449ee9ffa353ae6ea3d6a from peer 3b95f8c6589e473bb21db7681e227638 ({username='slave'} at 127.8.68.196:54853)
I20240501 20:09:52.156204 12253 tablet_copy_service.cc:161] P 18717c9754d04918815985cf07c1ddad: Beginning new tablet copy session on tablet 0faf9b33582449ee9ffa353ae6ea3d6a from peer 3b95f8c6589e473bb21db7681e227638 at {username='slave'} at 127.8.68.196:54853: session id = 3b95f8c6589e473bb21db7681e227638-0faf9b33582449ee9ffa353ae6ea3d6a
I20240501 20:09:52.158926 12253 tablet_copy_source_session.cc:205] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:52.163905 14502 ts_tablet_manager.cc:902] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:09:52.173990 14494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faf9b33582449ee9ffa353ae6ea3d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.197746 14465 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap replayed 1/1 log segments. Stats: ops{read=200 overwritten=0 applied=200 ignored=0} inserts{seen=881 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:52.198272 14465 tablet_bootstrap.cc:492] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: Bootstrap complete.
I20240501 20:09:52.198699 14465 ts_tablet_manager.cc:1367] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: Time spent bootstrapping tablet: real 0.440s	user 0.246s	sys 0.031s
I20240501 20:09:52.199728 14465 raft_consensus.cc:348] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:52.200178 14465 raft_consensus.cc:729] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 463bd46a1f6e4e89a3040c4cf50dd919, State: Initialized, Role: LEARNER
I20240501 20:09:52.200433 14465 consensus_queue.cc:260] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 200, Last appended: 2.200, Last appended by leader: 200, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: true } }
I20240501 20:09:52.204439 14465 ts_tablet_manager.cc:1398] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 20:09:52.207320 11993 tablet_copy_service.cc:342] P dc71b180f72b4c4697d28b13b363d705: Request end of tablet copy session 463bd46a1f6e4e89a3040c4cf50dd919-9d7dd13822784e789d558b4cc6c5010e received from {username='slave'} at 127.8.68.198:59931
I20240501 20:09:52.207439 11993 tablet_copy_service.cc:433] P dc71b180f72b4c4697d28b13b363d705: ending tablet copy session 463bd46a1f6e4e89a3040c4cf50dd919-9d7dd13822784e789d558b4cc6c5010e on tablet 9d7dd13822784e789d558b4cc6c5010e with peer 463bd46a1f6e4e89a3040c4cf50dd919
I20240501 20:09:52.263170 14494 tablet_copy_client.cc:783] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:52.283712 14494 tablet_copy_client.cc:647] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:52.289853 14494 tablet_copy_client.cc:523] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:52.298812 14494 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: Bootstrap starting.
I20240501 20:09:52.451961 14502 tablet_copy_client.cc:308] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:09:52.452767 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet 8baf6913f38a42c3a369cd673ff9438d from peer 535367472879403fa999dd00a155f042 ({username='slave'} at 127.8.68.197:35537)
I20240501 20:09:52.452913 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet 8baf6913f38a42c3a369cd673ff9438d from peer 535367472879403fa999dd00a155f042 at {username='slave'} at 127.8.68.197:35537: session id = 535367472879403fa999dd00a155f042-8baf6913f38a42c3a369cd673ff9438d
I20240501 20:09:52.455829 12123 tablet_copy_source_session.cc:205] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:52.457345 14502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baf6913f38a42c3a369cd673ff9438d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.465549 14502 tablet_copy_client.cc:783] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:52.465855 14502 tablet_copy_client.cc:647] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:52.470562 14502 tablet_copy_client.cc:523] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:52.474692 14502 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: Bootstrap starting.
I20240501 20:09:52.778873 13539 raft_consensus.cc:1203] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.185->[2.186-2.220]   Dedup: 2.200->[2.201-2.220]
I20240501 20:09:52.940379 14494 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: Bootstrap replayed 1/1 log segments. Stats: ops{read=239 overwritten=0 applied=239 ignored=0} inserts{seen=908 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:52.940960 14494 tablet_bootstrap.cc:492] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: Bootstrap complete.
I20240501 20:09:52.941344 14494 ts_tablet_manager.cc:1367] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: Time spent bootstrapping tablet: real 0.643s	user 0.274s	sys 0.037s
I20240501 20:09:52.946841 14494 raft_consensus.cc:348] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:52.947219 14494 raft_consensus.cc:729] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b95f8c6589e473bb21db7681e227638, State: Initialized, Role: LEARNER
I20240501 20:09:52.947474 14494 consensus_queue.cc:260] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 239, Last appended: 1.239, Last appended by leader: 239, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }
I20240501 20:09:52.977639 14511 raft_consensus.cc:1050] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: attempting to promote NON_VOTER 463bd46a1f6e4e89a3040c4cf50dd919 to VOTER
I20240501 20:09:52.978302 14511 consensus_queue.cc:237] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 2.227, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:52.985235 13275 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20240501 20:09:52.986124 14051 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20240501 20:09:52.989167 13537 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 LEARNER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20240501 20:09:52.990005 14403 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20240501 20:09:52.990675 14403 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Committing config change with OpId 2.228: config changed from index 186 to 228, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:52.995057 13275 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 186 to 228, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:52.996927 11827 catalog_manager.cc:5484] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index 186 to 228, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:53.003170 13539 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 186 to 228, 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:53.033017 14494 ts_tablet_manager.cc:1398] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638: Time spent starting tablet: real 0.091s	user 0.030s	sys 0.031s
I20240501 20:09:53.033849 12253 tablet_copy_service.cc:342] P 18717c9754d04918815985cf07c1ddad: Request end of tablet copy session 3b95f8c6589e473bb21db7681e227638-0faf9b33582449ee9ffa353ae6ea3d6a received from {username='slave'} at 127.8.68.196:54853
I20240501 20:09:53.033973 12253 tablet_copy_service.cc:433] P 18717c9754d04918815985cf07c1ddad: ending tablet copy session 3b95f8c6589e473bb21db7681e227638-0faf9b33582449ee9ffa353ae6ea3d6a on tablet 0faf9b33582449ee9ffa353ae6ea3d6a with peer 3b95f8c6589e473bb21db7681e227638
I20240501 20:09:53.282136 14502 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: Bootstrap replayed 1/1 log segments. Stats: ops{read=241 overwritten=0 applied=241 ignored=0} inserts{seen=869 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:53.283871 14502 tablet_bootstrap.cc:492] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: Bootstrap complete.
I20240501 20:09:53.284261 14502 ts_tablet_manager.cc:1367] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: Time spent bootstrapping tablet: real 0.810s	user 0.298s	sys 0.024s
I20240501 20:09:53.285254 14502 raft_consensus.cc:348] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:53.285566 14502 raft_consensus.cc:729] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 535367472879403fa999dd00a155f042, State: Initialized, Role: LEARNER
I20240501 20:09:53.285837 14502 consensus_queue.cc:260] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 241, Last appended: 1.241, Last appended by leader: 241, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: true } }
I20240501 20:09:53.286569 14502 ts_tablet_manager.cc:1398] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:09:53.287230 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 535367472879403fa999dd00a155f042-8baf6913f38a42c3a369cd673ff9438d received from {username='slave'} at 127.8.68.197:35537
I20240501 20:09:53.287353 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 535367472879403fa999dd00a155f042-8baf6913f38a42c3a369cd673ff9438d on tablet 8baf6913f38a42c3a369cd673ff9438d with peer 535367472879403fa999dd00a155f042
I20240501 20:09:53.409145 13276 raft_consensus.cc:1203] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.219->[1.220-1.270]   Dedup: 1.239->[1.240-1.270]
I20240501 20:09:53.592039 11969 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e"
dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54026
I20240501 20:09:53.592252 11969 raft_consensus.cc:595] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Received request to transfer leadership
W20240501 20:09:53.602128 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.606850 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.612686 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
I20240501 20:09:53.618908 14566 raft_consensus.cc:1050] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: attempting to promote NON_VOTER 3b95f8c6589e473bb21db7681e227638 to VOTER
W20240501 20:09:53.619628 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
I20240501 20:09:53.622917 14191 raft_consensus.cc:1050] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: attempting to promote NON_VOTER 3b95f8c6589e473bb21db7681e227638 to VOTER
I20240501 20:09:53.625797 14566 consensus_queue.cc:237] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 1.274, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
W20240501 20:09:53.627691 14191 raft_consensus.cc:1054] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: Unable to promote non-voter 3b95f8c6589e473bb21db7681e227638: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: true } }.
  Pending config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:53.628870 13275 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 1 LEARNER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 275. (index mismatch)
W20240501 20:09:53.629531 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
I20240501 20:09:53.632648 14564 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 273, Time since last communication: 0.000s
I20240501 20:09:53.635910 13274 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 220 to 275, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:53.636518 13408 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Refusing update from remote peer 18717c9754d04918815985cf07c1ddad: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 275. (index mismatch)
W20240501 20:09:53.638312 11932 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
I20240501 20:09:53.642654 11828 catalog_manager.cc:5484] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 reported cstate change: config changed from index 220 to 275, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "18717c9754d04918815985cf07c1ddad" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:53.644526 14192 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Committing config change with OpId 1.275: config changed from index 220 to 275, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:53.646457 14191 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 273, Time since last communication: 0.000s
I20240501 20:09:53.651844 13408 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 220 to 275, 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
W20240501 20:09:53.657042 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.674959 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
I20240501 20:09:53.685801 13408 raft_consensus.cc:1203] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.221->[1.222-1.272]   Dedup: 1.241->[1.242-1.272]
W20240501 20:09:53.691572 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.707574 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.725162 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.738160 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.753584 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.773746 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
I20240501 20:09:53.774709 14189 raft_consensus.cc:1050] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 535367472879403fa999dd00a155f042 to VOTER
I20240501 20:09:53.775277 14189 consensus_queue.cc:237] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 1.274, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:53.780880 13407 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 273. Preceding OpId from leader: term: 1 index: 275. (index mismatch)
I20240501 20:09:53.781499 13275 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 275. (index mismatch)
I20240501 20:09:53.782130 14329 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 273, Time since last communication: 0.000s
I20240501 20:09:53.784709 14189 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 272, Time since last communication: 0.000s
I20240501 20:09:53.784647 14092 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.275: config changed from index 222 to 275, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
W20240501 20:09:53.789418 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
I20240501 20:09:53.789873 11827 catalog_manager.cc:5484] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 222 to 275, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:53.792227 13274 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 222 to 275, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:53.798175 13406 raft_consensus.cc:1203] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.273->[1.274-1.275]   Dedup: 1.274->[1.275-1.275]
I20240501 20:09:53.803875 13406 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 222 to 275, 535367472879403fa999dd00a155f042 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
W20240501 20:09:53.810416 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.842495 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.862485 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.889818 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.917608 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.944106 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
W20240501 20:09:53.968860 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
I20240501 20:09:53.989336 14051 raft_consensus.cc:979] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: : Instructing follower 463bd46a1f6e4e89a3040c4cf50dd919 to start an election
I20240501 20:09:53.989495 14051 raft_consensus.cc:1067] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Signalling peer 463bd46a1f6e4e89a3040c4cf50dd919 to start an election
I20240501 20:09:53.992741 13538 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e"
dest_uuid: "463bd46a1f6e4e89a3040c4cf50dd919"
 from {username='slave'} at 127.8.68.193:49599
I20240501 20:09:53.992961 13538 raft_consensus.cc:482] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:53.993083 13538 raft_consensus.cc:3037] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:53.995970 13538 raft_consensus.cc:504] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:53.996771 13538 leader_election.cc:288] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [CANDIDATE]: Term 3 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397), 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351)
W20240501 20:09:54.000249 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Service unavailable: leader transfer in progress
I20240501 20:09:54.000980 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e" candidate_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" candidate_term: 3 candidate_status { last_received { term: 2 index: 235 } } ignore_live_leader: true dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:09:54.001178 11969 raft_consensus.cc:3032] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:54.001272 11969 raft_consensus.cc:729] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Running, Role: LEADER
I20240501 20:09:54.001511 11969 consensus_queue.cc:260] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 2.235, Last appended by leader: 235, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:54.001845 11969 raft_consensus.cc:3037] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:54.004025 12102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8baf6913f38a42c3a369cd673ff9438d"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59398
I20240501 20:09:54.004199 12102 raft_consensus.cc:595] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:54.004534 11969 raft_consensus.cc:2454] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 463bd46a1f6e4e89a3040c4cf50dd919 in term 3.
I20240501 20:09:54.004971 13471 leader_election.cc:302] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 463bd46a1f6e4e89a3040c4cf50dd919, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:09:54.005339 14317 raft_consensus.cc:2781] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:54.005481 14317 raft_consensus.cc:686] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 3 LEADER]: Becoming Leader. State: Replica: 463bd46a1f6e4e89a3040c4cf50dd919, State: Running, Role: LEADER
I20240501 20:09:54.005789 14317 consensus_queue.cc:237] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 2.235, Last appended by leader: 235, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
I20240501 20:09:54.009002 11827 catalog_manager.cc:5484] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 reported cstate change: term changed from 2 to 3, leader changed from dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) to 463bd46a1f6e4e89a3040c4cf50dd919 (127.8.68.198). New cstate: current_term: 3 leader_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:54.026983 11933 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40204: Illegal state: replica dc71b180f72b4c4697d28b13b363d705 is not leader of this config: current role FOLLOWER
I20240501 20:09:54.035657 13276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7dd13822784e789d558b4cc6c5010e" candidate_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" candidate_term: 3 candidate_status { last_received { term: 2 index: 235 } } ignore_live_leader: true dest_uuid: "3b95f8c6589e473bb21db7681e227638"
I20240501 20:09:54.035862 13276 raft_consensus.cc:3037] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:54.038308 13276 raft_consensus.cc:2454] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 463bd46a1f6e4e89a3040c4cf50dd919 in term 3.
W20240501 20:09:54.039273 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.044824 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.054551 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.064476 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.074389 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.085770 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.096535 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
I20240501 20:09:54.104285 13275 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 3 FOLLOWER]: Refusing update from remote peer 463bd46a1f6e4e89a3040c4cf50dd919: Log matching property violated. Preceding OpId in replica: term: 2 index: 235. Preceding OpId from leader: term: 3 index: 237. (index mismatch)
I20240501 20:09:54.104432 11972 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 3 FOLLOWER]: Refusing update from remote peer 463bd46a1f6e4e89a3040c4cf50dd919: Log matching property violated. Preceding OpId in replica: term: 2 index: 235. Preceding OpId from leader: term: 3 index: 237. (index mismatch)
I20240501 20:09:54.105114 14291 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240501 20:09:54.105865 14291 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
W20240501 20:09:54.107848 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
I20240501 20:09:54.108866 14525 mvcc.cc:205] Tried to move back new op lower bound from 7022977819045654528 to 7022977818650558464. Current Snapshot: MvccSnapshot[applied={T|T < 7022977819045654528}]
I20240501 20:09:54.120548 13539 consensus_queue.cc:237] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 3.237, Last appended by leader: 235, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } }
W20240501 20:09:54.125177 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
I20240501 20:09:54.125355 13275 raft_consensus.cc:1261] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 3 FOLLOWER]: Refusing update from remote peer 463bd46a1f6e4e89a3040c4cf50dd919: Log matching property violated. Preceding OpId in replica: term: 3 index: 237. Preceding OpId from leader: term: 3 index: 238. (index mismatch)
I20240501 20:09:54.126165 14291 consensus_queue.cc:1035] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240501 20:09:54.130180 13275 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 [term 3 FOLLOWER]: Committing config change with OpId 3.238: config changed from index 228 to 238, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:54.135452 14317 raft_consensus.cc:2932] T 9d7dd13822784e789d558b4cc6c5010e P 463bd46a1f6e4e89a3040c4cf50dd919 [term 3 LEADER]: Committing config change with OpId 3.238: config changed from index 228 to 238, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
I20240501 20:09:54.135804 11827 catalog_manager.cc:5484] T 9d7dd13822784e789d558b4cc6c5010e P 3b95f8c6589e473bb21db7681e227638 reported cstate change: config changed from index 228 to 238, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New cstate: current_term: 3 leader_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "463bd46a1f6e4e89a3040c4cf50dd919" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 35531 } attrs { promote: false } } }
W20240501 20:09:54.138670 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
I20240501 20:09:54.140398 11819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d7dd13822784e789d558b4cc6c5010e with cas_config_opid_index 228: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:54.141744 11952 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7dd13822784e789d558b4cc6c5010e with delete_type TABLET_DATA_TOMBSTONED (TS dc71b180f72b4c4697d28b13b363d705 not found in new config with opid_index 238) from {username='slave'} at 127.0.0.1:58526
W20240501 20:09:54.154172 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
I20240501 20:09:54.157896 14603 tablet_replica.cc:330] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: stopping tablet replica
I20240501 20:09:54.158169 14603 raft_consensus.cc:2229] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:54.159031 14603 raft_consensus.cc:2258] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:54.160969 14603 ts_tablet_manager.cc:1875] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:54.174244 14603 ts_tablet_manager.cc:1888] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.237
I20240501 20:09:54.174502 14603 log.cc:1196] T 9d7dd13822784e789d558b4cc6c5010e P dc71b180f72b4c4697d28b13b363d705: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/9d7dd13822784e789d558b4cc6c5010e
I20240501 20:09:54.175691 11816 catalog_manager.cc:4830] TS dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397): tablet 9d7dd13822784e789d558b4cc6c5010e (table rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]) successfully deleted
W20240501 20:09:54.175823 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.191785 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.210780 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.231648 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.250429 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
I20240501 20:09:54.256491 12231 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0faf9b33582449ee9ffa353ae6ea3d6a"
dest_uuid: "18717c9754d04918815985cf07c1ddad"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45922
I20240501 20:09:54.256750 12231 raft_consensus.cc:595] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Received request to transfer leadership
W20240501 20:09:54.273253 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.279050 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.284051 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.293534 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.296380 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.302579 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.313371 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.318235 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.321491 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.330132 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.341429 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.342871 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.351090 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.363822 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.367038 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.377353 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.389972 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
W20240501 20:09:54.393687 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
I20240501 20:09:54.409593 14092 raft_consensus.cc:979] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: : Instructing follower 535367472879403fa999dd00a155f042 to start an election
I20240501 20:09:54.409716 14092 raft_consensus.cc:1067] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Signalling peer 535367472879403fa999dd00a155f042 to start an election
I20240501 20:09:54.411563 13406 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8baf6913f38a42c3a369cd673ff9438d"
dest_uuid: "535367472879403fa999dd00a155f042"
 from {username='slave'} at 127.8.68.194:42397
I20240501 20:09:54.411772 13406 raft_consensus.cc:482] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:54.411880 13406 raft_consensus.cc:3037] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:09:54.413219 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.417048 12063 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Service unavailable: leader transfer in progress
I20240501 20:09:54.414544 13406 raft_consensus.cc:504] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:54.418751 12102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf6913f38a42c3a369cd673ff9438d" candidate_uuid: "535367472879403fa999dd00a155f042" candidate_term: 2 candidate_status { last_received { term: 1 index: 281 } } ignore_live_leader: true dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
I20240501 20:09:54.418972 12102 raft_consensus.cc:3032] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:54.419059 12102 raft_consensus.cc:729] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:09:54.419377 12102 consensus_queue.cc:260] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 281, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:54.419721 12102 raft_consensus.cc:3037] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:54.420444 13406 leader_election.cc:288] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [CANDIDATE]: Term 2 election: Requested vote from peers 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853), 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351)
I20240501 20:09:54.422155 12102 raft_consensus.cc:2454] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 535367472879403fa999dd00a155f042 in term 2.
I20240501 20:09:54.427688 12103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8baf6913f38a42c3a369cd673ff9438d"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59410
I20240501 20:09:54.427852 12103 raft_consensus.cc:595] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Received request to transfer leadership
I20240501 20:09:54.427938 12103 raft_consensus.cc:603] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 20:09:54.429702 13275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf6913f38a42c3a369cd673ff9438d" candidate_uuid: "535367472879403fa999dd00a155f042" candidate_term: 2 candidate_status { last_received { term: 1 index: 281 } } ignore_live_leader: true dest_uuid: "3b95f8c6589e473bb21db7681e227638"
I20240501 20:09:54.429879 13275 raft_consensus.cc:3037] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:09:54.431361 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
I20240501 20:09:54.431396 13343 leader_election.cc:302] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, 535367472879403fa999dd00a155f042; no voters: 
I20240501 20:09:54.431675 14335 raft_consensus.cc:2781] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:54.431784 14335 raft_consensus.cc:686] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 2 LEADER]: Becoming Leader. State: Replica: 535367472879403fa999dd00a155f042, State: Running, Role: LEADER
I20240501 20:09:54.432044 14335 consensus_queue.cc:237] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 281, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:54.432222 13275 raft_consensus.cc:2454] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 535367472879403fa999dd00a155f042 in term 2.
I20240501 20:09:54.435222 11827 catalog_manager.cc:5484] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 reported cstate change: term changed from 1 to 2, leader changed from 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) to 535367472879403fa999dd00a155f042 (127.8.68.197). New cstate: current_term: 2 leader_uuid: "535367472879403fa999dd00a155f042" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:54.443660 12062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56360: Illegal state: replica 34f8af7f2d4245899a7d8fd5012a3721 is not leader of this config: current role FOLLOWER
W20240501 20:09:54.451231 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.471738 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.494541 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.517311 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.540498 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
I20240501 20:09:54.563200 13274 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer 535367472879403fa999dd00a155f042: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 2 index: 283. (index mismatch)
I20240501 20:09:54.563225 12102 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Refusing update from remote peer 535367472879403fa999dd00a155f042: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 2 index: 283. (index mismatch)
I20240501 20:09:54.563799 14335 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240501 20:09:54.564291 14335 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
W20240501 20:09:54.565492 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
I20240501 20:09:54.566318 12232 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0faf9b33582449ee9ffa353ae6ea3d6a"
dest_uuid: "18717c9754d04918815985cf07c1ddad"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45942
I20240501 20:09:54.566495 12232 raft_consensus.cc:595] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:54.587987 13406 consensus_queue.cc:237] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 2.283, Last appended by leader: 281, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } }
I20240501 20:09:54.589498 13275 raft_consensus.cc:1261] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer 535367472879403fa999dd00a155f042: Log matching property violated. Preceding OpId in replica: term: 2 index: 283. Preceding OpId from leader: term: 2 index: 284. (index mismatch)
I20240501 20:09:54.589999 14332 consensus_queue.cc:1035] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
W20240501 20:09:54.591239 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
I20240501 20:09:54.598668 14643 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 535367472879403fa999dd00a155f042 [term 2 LEADER]: Committing config change with OpId 2.284: config changed from index 275 to 284, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:54.601272 13275 raft_consensus.cc:2932] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Committing config change with OpId 2.284: config changed from index 275 to 284, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:54.602814 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8baf6913f38a42c3a369cd673ff9438d with cas_config_opid_index 275: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:54.606755 11828 catalog_manager.cc:5484] T 8baf6913f38a42c3a369cd673ff9438d P 3b95f8c6589e473bb21db7681e227638 reported cstate change: config changed from index 275 to 284, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "535367472879403fa999dd00a155f042" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } }
I20240501 20:09:54.611435 12083 tablet_service.cc:1514] Processing DeleteTablet for tablet 8baf6913f38a42c3a369cd673ff9438d with delete_type TABLET_DATA_TOMBSTONED (TS 34f8af7f2d4245899a7d8fd5012a3721 not found in new config with opid_index 284) from {username='slave'} at 127.0.0.1:42804
I20240501 20:09:54.612092 14644 tablet_replica.cc:330] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: stopping tablet replica
I20240501 20:09:54.612279 14644 raft_consensus.cc:2229] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:54.612589 14644 raft_consensus.cc:2258] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:54.613571 14644 ts_tablet_manager.cc:1875] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:09:54.615859 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
I20240501 20:09:54.621519 14644 ts_tablet_manager.cc:1888] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.283
I20240501 20:09:54.621686 14644 log.cc:1196] T 8baf6913f38a42c3a369cd673ff9438d P 34f8af7f2d4245899a7d8fd5012a3721: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/8baf6913f38a42c3a369cd673ff9438d
I20240501 20:09:54.622807 11818 catalog_manager.cc:4830] TS 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853): tablet 8baf6913f38a42c3a369cd673ff9438d (table rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]) successfully deleted
W20240501 20:09:54.643381 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.671916 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
W20240501 20:09:54.704198 12193 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Service unavailable: leader transfer in progress
I20240501 20:09:54.721907 14191 raft_consensus.cc:979] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: : Instructing follower 3b95f8c6589e473bb21db7681e227638 to start an election
I20240501 20:09:54.722229 14564 raft_consensus.cc:1067] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Signalling peer 3b95f8c6589e473bb21db7681e227638 to start an election
I20240501 20:09:54.723172 13274 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0faf9b33582449ee9ffa353ae6ea3d6a"
dest_uuid: "3b95f8c6589e473bb21db7681e227638"
 from {username='slave'} at 127.8.68.195:55423
I20240501 20:09:54.723441 13274 raft_consensus.cc:482] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:54.723605 13274 raft_consensus.cc:3037] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:54.726176 13274 raft_consensus.cc:504] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:54.729089 13274 leader_election.cc:288] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [CANDIDATE]: Term 2 election: Requested vote from peers 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201), 535367472879403fa999dd00a155f042 (127.8.68.197:43403)
I20240501 20:09:54.729569 12231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faf9b33582449ee9ffa353ae6ea3d6a" candidate_uuid: "3b95f8c6589e473bb21db7681e227638" candidate_term: 2 candidate_status { last_received { term: 1 index: 293 } } ignore_live_leader: true dest_uuid: "18717c9754d04918815985cf07c1ddad"
I20240501 20:09:54.729766 12231 raft_consensus.cc:3032] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:54.729874 12231 raft_consensus.cc:729] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18717c9754d04918815985cf07c1ddad, State: Running, Role: LEADER
I20240501 20:09:54.730176 12231 consensus_queue.cc:260] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 293, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:54.730630 12231 raft_consensus.cc:3037] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:54.733983 12231 raft_consensus.cc:2454] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b95f8c6589e473bb21db7681e227638 in term 2.
W20240501 20:09:54.734834 12192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38662: Illegal state: replica 18717c9754d04918815985cf07c1ddad is not leader of this config: current role FOLLOWER
I20240501 20:09:54.735677 13210 leader_election.cc:302] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18717c9754d04918815985cf07c1ddad, 3b95f8c6589e473bb21db7681e227638; no voters: 
I20240501 20:09:54.736323 14534 raft_consensus.cc:2781] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:54.758543 14534 raft_consensus.cc:686] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 2 LEADER]: Becoming Leader. State: Replica: 3b95f8c6589e473bb21db7681e227638, State: Running, Role: LEADER
I20240501 20:09:54.759009 14534 consensus_queue.cc:237] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 293, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:54.763444 11827 catalog_manager.cc:5484] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 reported cstate change: term changed from 1 to 2, leader changed from 18717c9754d04918815985cf07c1ddad (127.8.68.195) to 3b95f8c6589e473bb21db7681e227638 (127.8.68.196). New cstate: current_term: 2 leader_uuid: "3b95f8c6589e473bb21db7681e227638" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:54.805796 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faf9b33582449ee9ffa353ae6ea3d6a" candidate_uuid: "3b95f8c6589e473bb21db7681e227638" candidate_term: 2 candidate_status { last_received { term: 1 index: 293 } } ignore_live_leader: true dest_uuid: "535367472879403fa999dd00a155f042"
I20240501 20:09:54.806123 13406 raft_consensus.cc:3037] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:54.808966 13406 raft_consensus.cc:2454] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b95f8c6589e473bb21db7681e227638 in term 2.
I20240501 20:09:54.880391 13406 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 2 FOLLOWER]: Refusing update from remote peer 3b95f8c6589e473bb21db7681e227638: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20240501 20:09:54.881031 14534 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:09:54.887245 12231 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 2 FOLLOWER]: Refusing update from remote peer 3b95f8c6589e473bb21db7681e227638: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20240501 20:09:54.888350 14647 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:09:54.992133 13274 consensus_queue.cc:237] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 2.296, Last appended by leader: 293, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } }
I20240501 20:09:54.998742 13408 raft_consensus.cc:1261] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 2 FOLLOWER]: Refusing update from remote peer 3b95f8c6589e473bb21db7681e227638: Log matching property violated. Preceding OpId in replica: term: 2 index: 296. Preceding OpId from leader: term: 2 index: 297. (index mismatch)
I20240501 20:09:55.000653 14664 consensus_queue.cc:1035] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20240501 20:09:55.003968 14647 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 [term 2 LEADER]: Committing config change with OpId 2.297: config changed from index 275 to 297, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:55.008545 13408 raft_consensus.cc:2932] T 0faf9b33582449ee9ffa353ae6ea3d6a P 535367472879403fa999dd00a155f042 [term 2 FOLLOWER]: Committing config change with OpId 2.297: config changed from index 275 to 297, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } }
I20240501 20:09:55.012231 11827 catalog_manager.cc:5484] T 0faf9b33582449ee9ffa353ae6ea3d6a P 3b95f8c6589e473bb21db7681e227638 reported cstate change: config changed from index 275 to 297, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New cstate: current_term: 2 leader_uuid: "3b95f8c6589e473bb21db7681e227638" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "535367472879403fa999dd00a155f042" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 43403 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:55.016505 12212 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faf9b33582449ee9ffa353ae6ea3d6a with delete_type TABLET_DATA_TOMBSTONED (TS 18717c9754d04918815985cf07c1ddad not found in new config with opid_index 297) from {username='slave'} at 127.0.0.1:50996
I20240501 20:09:55.017243 14669 tablet_replica.cc:330] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: stopping tablet replica
I20240501 20:09:55.017674 14669 raft_consensus.cc:2229] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:55.017982 14669 raft_consensus.cc:2258] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:55.019593 14669 ts_tablet_manager.cc:1875] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:55.022861 11819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0faf9b33582449ee9ffa353ae6ea3d6a with cas_config_opid_index 275: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:55.029076 14669 ts_tablet_manager.cc:1888] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.296
I20240501 20:09:55.029315 14669 log.cc:1196] T 0faf9b33582449ee9ffa353ae6ea3d6a P 18717c9754d04918815985cf07c1ddad: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/0faf9b33582449ee9ffa353ae6ea3d6a
I20240501 20:09:55.030381 11816 catalog_manager.cc:4830] TS 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201): tablet 0faf9b33582449ee9ffa353ae6ea3d6a (table rebalance_test_table_2 [id=b40a568bc0e8467881e6ef108d91304f]) successfully deleted
I20240501 20:09:55.821754 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:55.939613 12083 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:55.981559 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:55.999742 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:56.002861 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:56.036330 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:56.167222 11952 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:56.887408 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:56.951267 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:56.983954 12083 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:56.996811 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:57.079440 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:57.080135 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:57.148672 11952 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:59.049751 12083 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:09:59.060778 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:09:59.070211 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:59.109169 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:59.152524 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:59.172749 11952 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:59.267654 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:00.302819 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:00.315424 12083 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:00.342234 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:00.364063 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:00.393972 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:00.433979 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:00.446923 11952 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:01.152127 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:01.176302 12083 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:01.218158 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:01.220757 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:01.221239 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:01.226966 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:01.260063 11952 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 466d4986891e4af2a4e44757733bfd5c | 127.8.68.254:46165 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.8.68.212:43765 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 18717c9754d04918815985cf07c1ddad | 127.8.68.195:42201 | HEALTHY | <none>   |       3        |       0
 34f8af7f2d4245899a7d8fd5012a3721 | 127.8.68.194:32853 | HEALTHY | <none>   |       5        |       0
 3b95f8c6589e473bb21db7681e227638 | 127.8.68.196:36351 | HEALTHY | <none>   |       1        |       0
 463bd46a1f6e4e89a3040c4cf50dd919 | 127.8.68.198:35531 | HEALTHY | <none>   |       2        |       0
 535367472879403fa999dd00a155f042 | 127.8.68.197:43403 | HEALTHY | <none>   |       2        |       0
 aed522b44a8e420daf65718450541c3c | 127.8.68.199:40105 | HEALTHY | <none>   |       0        |       0
 dc71b180f72b4c4697d28b13b363d705 | 127.8.68.193:39397 | HEALTHY | <none>   |       3        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.8.68.193                                                                                                                                                    | experimental          | 127.8.68.193:39397
 local_ip_for_outbound_sockets            | 127.8.68.194                                                                                                                                                    | experimental          | 127.8.68.194:32853
 local_ip_for_outbound_sockets            | 127.8.68.195                                                                                                                                                    | experimental          | 127.8.68.195:42201
 local_ip_for_outbound_sockets            | 127.8.68.196                                                                                                                                                    | experimental          | 127.8.68.196:36351
 local_ip_for_outbound_sockets            | 127.8.68.197                                                                                                                                                    | experimental          | 127.8.68.197:43403
 local_ip_for_outbound_sockets            | 127.8.68.198                                                                                                                                                    | experimental          | 127.8.68.198:35531
 local_ip_for_outbound_sockets            | 127.8.68.199                                                                                                                                                    | experimental          | 127.8.68.199:40105
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                            | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.8.68.193:39397
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.8.68.194:32853
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.8.68.195:42201
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.8.68.196:36351
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.8.68.197:43403
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.8.68.198:35531
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.8.68.199:40105

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.8.68.212:43765 | all 7 server(s) checked
 time_source         | builtin            | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240501 20:10:01.309533  8467 log_verifier.cc:126] Checking tablet 0faf9b33582449ee9ffa353ae6ea3d6a
I20240501 20:10:01.477033  8467 log_verifier.cc:177] Verified matching terms for 435 ops in tablet 0faf9b33582449ee9ffa353ae6ea3d6a
I20240501 20:10:01.477819  8467 log_verifier.cc:126] Checking tablet 262853fe9d5c4aa7acdac809a6378567
I20240501 20:10:01.499148  8467 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 262853fe9d5c4aa7acdac809a6378567
I20240501 20:10:01.499265  8467 log_verifier.cc:126] Checking tablet 2c0a1865e31241569216a3b4b072829c
I20240501 20:10:01.699926  8467 log_verifier.cc:177] Verified matching terms for 509 ops in tablet 2c0a1865e31241569216a3b4b072829c
I20240501 20:10:01.700503  8467 log_verifier.cc:126] Checking tablet 321fb505be2f4263a79c9bcba4be6616
I20240501 20:10:01.855355  8467 log_verifier.cc:177] Verified matching terms for 376 ops in tablet 321fb505be2f4263a79c9bcba4be6616
I20240501 20:10:01.855811  8467 log_verifier.cc:126] Checking tablet 38b8e2865d134700ad0959372afc6670
I20240501 20:10:02.006142  8467 log_verifier.cc:177] Verified matching terms for 434 ops in tablet 38b8e2865d134700ad0959372afc6670
I20240501 20:10:02.006695  8467 log_verifier.cc:126] Checking tablet 4ad26a211a9c4c609da559d2681c837a
I20240501 20:10:02.182695  8467 log_verifier.cc:177] Verified matching terms for 444 ops in tablet 4ad26a211a9c4c609da559d2681c837a
I20240501 20:10:02.183300  8467 log_verifier.cc:126] Checking tablet 575f6e5055d84dc38ef2326cec06fa47
I20240501 20:10:02.335027  8467 log_verifier.cc:177] Verified matching terms for 396 ops in tablet 575f6e5055d84dc38ef2326cec06fa47
I20240501 20:10:02.335495  8467 log_verifier.cc:126] Checking tablet 67043e58e5fd4233bc945432eeae1227
I20240501 20:10:02.509042  8467 log_verifier.cc:177] Verified matching terms for 450 ops in tablet 67043e58e5fd4233bc945432eeae1227
I20240501 20:10:02.509559  8467 log_verifier.cc:126] Checking tablet 7eb828d8511f40c78004d0aaba843df5
I20240501 20:10:02.643630  8467 log_verifier.cc:177] Verified matching terms for 378 ops in tablet 7eb828d8511f40c78004d0aaba843df5
I20240501 20:10:02.644116  8467 log_verifier.cc:126] Checking tablet 8baf6913f38a42c3a369cd673ff9438d
I20240501 20:10:02.788215  8467 log_verifier.cc:177] Verified matching terms for 416 ops in tablet 8baf6913f38a42c3a369cd673ff9438d
I20240501 20:10:02.788733  8467 log_verifier.cc:126] Checking tablet 9d7dd13822784e789d558b4cc6c5010e
I20240501 20:10:02.920668  8467 log_verifier.cc:177] Verified matching terms for 375 ops in tablet 9d7dd13822784e789d558b4cc6c5010e
I20240501 20:10:02.921110  8467 log_verifier.cc:126] Checking tablet cf6c676aa51e44a9b8a1c2a3c828ccd0
I20240501 20:10:03.088660  8467 log_verifier.cc:177] Verified matching terms for 447 ops in tablet cf6c676aa51e44a9b8a1c2a3c828ccd0
I20240501 20:10:03.089236  8467 log_verifier.cc:126] Checking tablet d847215537454b4ba9e8a8b87ec91699
I20240501 20:10:03.248576  8467 log_verifier.cc:177] Verified matching terms for 466 ops in tablet d847215537454b4ba9e8a8b87ec91699
I20240501 20:10:03.249140  8467 log_verifier.cc:126] Checking tablet e97529f5c0244486b7d946ff4526fe59
I20240501 20:10:03.382771  8467 log_verifier.cc:177] Verified matching terms for 348 ops in tablet e97529f5c0244486b7d946ff4526fe59
I20240501 20:10:03.383206  8467 log_verifier.cc:126] Checking tablet f25dcb94e89c4c42979fdb39523b7fd2
I20240501 20:10:03.537611  8467 log_verifier.cc:177] Verified matching terms for 408 ops in tablet f25dcb94e89c4c42979fdb39523b7fd2
I20240501 20:10:03.538110  8467 log_verifier.cc:126] Checking tablet f8f9c24067b442299db6fa5831feb56d
I20240501 20:10:03.690930  8467 log_verifier.cc:177] Verified matching terms for 401 ops in tablet f8f9c24067b442299db6fa5831feb56d
I20240501 20:10:03.694244  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.200:0
--local_ip_for_outbound_sockets=127.8.68.200
--webserver_interface=127.8.68.200
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--builtin_ntp_servers=127.8.68.212:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:10:04.022329 14955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:04.022593 14955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:04.022722 14955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:04.030148 14955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:04.030258 14955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:04.030411 14955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.200
I20240501 20:10:04.039331 14955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.68.200
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.200
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:04.040385 14955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:04.041888 14955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:04.054088 14961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:04.054602 14962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:04.055022 14964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:04.055516 14955 server_base.cc:1034] running on GCE node
I20240501 20:10:04.056107 14955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:04.057071 14955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:04.058272 14955 hybrid_clock.cc:648] HybridClock initialized: now 1714594204058232 us; error 33 us; skew 500 ppm
I20240501 20:10:04.058588 14955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:04.060599 14955 webserver.cc:460] Webserver started at http://127.8.68.200:38929/ using document root <none> and password file <none>
I20240501 20:10:04.061250 14955 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:04.061352 14955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:04.061661 14955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:04.064100 14955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "8658f3ae7457494faaf046c3c088291d"
format_stamp: "Formatted at 2024-05-01 20:10:04 on dist-test-slave-08p9"
I20240501 20:10:04.064790 14955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "8658f3ae7457494faaf046c3c088291d"
format_stamp: "Formatted at 2024-05-01 20:10:04 on dist-test-slave-08p9"
I20240501 20:10:04.069309 14955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20240501 20:10:04.072376 14970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:04.073660 14955 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:10:04.073802 14955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8658f3ae7457494faaf046c3c088291d"
format_stamp: "Formatted at 2024-05-01 20:10:04 on dist-test-slave-08p9"
I20240501 20:10:04.073978 14955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:04.100025 14955 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:04.100747 14955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:04.101004 14955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:04.101861 14955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:04.103286 14955 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:04.103374 14955 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:04.103478 14955 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:04.103554 14955 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:04.131975 14955 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.200:40535
I20240501 20:10:04.132033 15082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.200:40535 every 8 connection(s)
I20240501 20:10:04.133141 14955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:10:04.141723  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 14955
I20240501 20:10:04.141925  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:10:04.145015 15083 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46165
I20240501 20:10:04.145310 15083 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:04.146173 15083 heartbeater.cc:511] Master 127.8.68.254:46165 requested a full tablet report, sending...
I20240501 20:10:04.147518 11829 ts_manager.cc:194] Registered new tserver with Master: 8658f3ae7457494faaf046c3c088291d (127.8.68.200:40535)
I20240501 20:10:04.148474 11829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.200:52317
I20240501 20:10:04.601827 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:04.615607 12083 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:04.640619 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:04.640851 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:04.650779 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:04.698989 11952 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:04.699501 15018 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:04.712827 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:04.864069 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:04.882123 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:04.883086 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:04.888867 12083 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:10:04.926399 15018 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:04.930599 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:04.952934 13649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:04.962702 11952 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:05.017794 11972 consensus_queue.cc:237] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 401, Committed index: 401, Last appended: 1.401, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:05.020428 13667 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch)
I20240501 20:10:05.020969 15177 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 401, Time since last communication: 0.000s
I20240501 20:10:05.023675 14868 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.402: config changed from index 139 to 402, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } } }
I20240501 20:10:05.024287 13667 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 139 to 402, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } } }
I20240501 20:10:05.028857 11826 catalog_manager.cc:5484] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index 139 to 402, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:05.029331 11906 consensus_peers.cc:485] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 -> Peer 8658f3ae7457494faaf046c3c088291d (127.8.68.200:40535): Couldn't send request to peer 8658f3ae7457494faaf046c3c088291d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8f9c24067b442299db6fa5831feb56d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:05.040526 12102 consensus_queue.cc:237] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 1.378, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:05.042706 13275 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 378. Preceding OpId from leader: term: 1 index: 379. (index mismatch)
I20240501 20:10:05.043597 14772 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 379, Last known committed idx: 378, Time since last communication: 0.000s
I20240501 20:10:05.046504 15191 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.379: config changed from index 98 to 379, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } } }
I20240501 20:10:05.047086 13275 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.379: config changed from index 98 to 379, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } } }
W20240501 20:10:05.049496 12036 consensus_peers.cc:485] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 8658f3ae7457494faaf046c3c088291d (127.8.68.200:40535): Couldn't send request to peer 8658f3ae7457494faaf046c3c088291d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7eb828d8511f40c78004d0aaba843df5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:05.052328 11826 catalog_manager.cc:5484] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 reported cstate change: config changed from index 98 to 379, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } } }
I20240501 20:10:05.070252 12102 consensus_queue.cc:237] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 1.348, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
W20240501 20:10:05.072101 12036 consensus_peers.cc:485] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 8658f3ae7457494faaf046c3c088291d (127.8.68.200:40535): Couldn't send request to peer 8658f3ae7457494faaf046c3c088291d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e97529f5c0244486b7d946ff4526fe59. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:05.072381 12232 raft_consensus.cc:1261] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20240501 20:10:05.072834 15191 consensus_queue.cc:1035] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20240501 20:10:05.075086 14772 raft_consensus.cc:2932] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.349: config changed from index -1 to 349, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } } }
I20240501 20:10:05.078392 12232 raft_consensus.cc:2932] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index -1 to 349, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } } }
I20240501 20:10:05.079365 11826 catalog_manager.cc:5484] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index -1 to 349, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:05.090860 12102 consensus_queue.cc:237] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
W20240501 20:10:05.092597 12036 consensus_peers.cc:485] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 -> Peer 8658f3ae7457494faaf046c3c088291d (127.8.68.200:40535): Couldn't send request to peer 8658f3ae7457494faaf046c3c088291d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 262853fe9d5c4aa7acdac809a6378567. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:05.092780 13275 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:05.093277 15201 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:05.095888 15191 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } } }
I20240501 20:10:05.096467 13275 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } } }
I20240501 20:10:05.099663 11825 catalog_manager.cc:5484] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 5 to 6, NON_VOTER 8658f3ae7457494faaf046c3c088291d (127.8.68.200) added. New cstate: current_term: 2 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:05.151073 15083 heartbeater.cc:503] Master 127.8.68.254:46165 was elected leader, sending a full tablet report...
I20240501 20:10:05.531276 15249 ts_tablet_manager.cc:902] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: Initiating tablet copy from peer dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397)
I20240501 20:10:05.532895 15249 tablet_copy_client.cc:308] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:39397
I20240501 20:10:05.538723 11993 tablet_copy_service.cc:140] P dc71b180f72b4c4697d28b13b363d705: Received BeginTabletCopySession request for tablet f8f9c24067b442299db6fa5831feb56d from peer 8658f3ae7457494faaf046c3c088291d ({username='slave'} at 127.8.68.200:56713)
I20240501 20:10:05.538863 11993 tablet_copy_service.cc:161] P dc71b180f72b4c4697d28b13b363d705: Beginning new tablet copy session on tablet f8f9c24067b442299db6fa5831feb56d from peer 8658f3ae7457494faaf046c3c088291d at {username='slave'} at 127.8.68.200:56713: session id = 8658f3ae7457494faaf046c3c088291d-f8f9c24067b442299db6fa5831feb56d
I20240501 20:10:05.540791 11993 tablet_copy_source_session.cc:205] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:05.544761 15249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f9c24067b442299db6fa5831feb56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.553370 15249 tablet_copy_client.cc:783] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:05.553632 15249 tablet_copy_client.cc:647] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:05.556143 15249 tablet_copy_client.cc:523] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:05.558892 15249 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: Bootstrap starting.
I20240501 20:10:05.595732 15265 ts_tablet_manager.cc:902] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:10:05.611855 15265 tablet_copy_client.cc:308] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:10:05.619479 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet 262853fe9d5c4aa7acdac809a6378567 from peer 8658f3ae7457494faaf046c3c088291d ({username='slave'} at 127.8.68.200:57893)
I20240501 20:10:05.619628 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet 262853fe9d5c4aa7acdac809a6378567 from peer 8658f3ae7457494faaf046c3c088291d at {username='slave'} at 127.8.68.200:57893: session id = 8658f3ae7457494faaf046c3c088291d-262853fe9d5c4aa7acdac809a6378567
I20240501 20:10:05.622112 12123 tablet_copy_source_session.cc:205] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:05.623474 15265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 262853fe9d5c4aa7acdac809a6378567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.630043 15265 tablet_copy_client.cc:783] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:05.630375 15265 tablet_copy_client.cc:647] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:05.631484 15249 log.cc:824] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:05.631856 15265 tablet_copy_client.cc:523] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:05.634506 15265 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: Bootstrap starting.
I20240501 20:10:05.641047 15265 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:05.641463 15265 tablet_bootstrap.cc:492] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: Bootstrap complete.
I20240501 20:10:05.641819 15265 ts_tablet_manager.cc:1367] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.001s
I20240501 20:10:05.646454 15265 raft_consensus.cc:348] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:05.646832 15265 raft_consensus.cc:729] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8658f3ae7457494faaf046c3c088291d, State: Initialized, Role: LEARNER
I20240501 20:10:05.647302 15265 consensus_queue.cc:260] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:05.650044 15265 ts_tablet_manager.cc:1398] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.005s
I20240501 20:10:05.651394 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 8658f3ae7457494faaf046c3c088291d-262853fe9d5c4aa7acdac809a6378567 received from {username='slave'} at 127.8.68.200:57893
I20240501 20:10:05.651500 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 8658f3ae7457494faaf046c3c088291d-262853fe9d5c4aa7acdac809a6378567 on tablet 262853fe9d5c4aa7acdac809a6378567 with peer 8658f3ae7457494faaf046c3c088291d
I20240501 20:10:05.662652 15265 ts_tablet_manager.cc:902] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:10:05.663795 15265 tablet_copy_client.cc:308] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:10:05.664410 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet 7eb828d8511f40c78004d0aaba843df5 from peer 8658f3ae7457494faaf046c3c088291d ({username='slave'} at 127.8.68.200:57893)
I20240501 20:10:05.664573 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet 7eb828d8511f40c78004d0aaba843df5 from peer 8658f3ae7457494faaf046c3c088291d at {username='slave'} at 127.8.68.200:57893: session id = 8658f3ae7457494faaf046c3c088291d-7eb828d8511f40c78004d0aaba843df5
I20240501 20:10:05.667409 12123 tablet_copy_source_session.cc:205] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:05.668527 15265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb828d8511f40c78004d0aaba843df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.668591 15287 ts_tablet_manager.cc:902] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: Initiating tablet copy from peer 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853)
I20240501 20:10:05.678951 15265 tablet_copy_client.cc:783] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:05.679255 15265 tablet_copy_client.cc:647] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:05.680115 15287 tablet_copy_client.cc:308] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:32853
I20240501 20:10:05.681241 12123 tablet_copy_service.cc:140] P 34f8af7f2d4245899a7d8fd5012a3721: Received BeginTabletCopySession request for tablet e97529f5c0244486b7d946ff4526fe59 from peer 8658f3ae7457494faaf046c3c088291d ({username='slave'} at 127.8.68.200:57893)
I20240501 20:10:05.681375 12123 tablet_copy_service.cc:161] P 34f8af7f2d4245899a7d8fd5012a3721: Beginning new tablet copy session on tablet e97529f5c0244486b7d946ff4526fe59 from peer 8658f3ae7457494faaf046c3c088291d at {username='slave'} at 127.8.68.200:57893: session id = 8658f3ae7457494faaf046c3c088291d-e97529f5c0244486b7d946ff4526fe59
I20240501 20:10:05.681968 15265 tablet_copy_client.cc:523] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:05.683362 12123 tablet_copy_source_session.cc:205] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:05.684249 15287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97529f5c0244486b7d946ff4526fe59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.685223 15265 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: Bootstrap starting.
I20240501 20:10:05.688908 15287 tablet_copy_client.cc:783] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:05.689194 15287 tablet_copy_client.cc:647] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:05.694137 15287 tablet_copy_client.cc:523] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:05.696635 15287 tablet_bootstrap.cc:492] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: Bootstrap starting.
I20240501 20:10:06.063704 15038 raft_consensus.cc:1203] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6]   Dedup: 2.6->[]
I20240501 20:10:06.110400 15287 tablet_bootstrap.cc:492] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: Bootstrap replayed 1/1 log segments. Stats: ops{read=349 overwritten=0 applied=349 ignored=0} inserts{seen=980 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:06.110910 15287 tablet_bootstrap.cc:492] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: Bootstrap complete.
I20240501 20:10:06.111250 15287 ts_tablet_manager.cc:1367] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: Time spent bootstrapping tablet: real 0.415s	user 0.373s	sys 0.021s
I20240501 20:10:06.112327 15287 raft_consensus.cc:348] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:06.112610 15287 raft_consensus.cc:729] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8658f3ae7457494faaf046c3c088291d, State: Initialized, Role: LEARNER
I20240501 20:10:06.112856 15287 consensus_queue.cc:260] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 349, Last appended: 1.349, Last appended by leader: 349, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:06.113514 15287 ts_tablet_manager.cc:1398] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:10:06.114262 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 8658f3ae7457494faaf046c3c088291d-e97529f5c0244486b7d946ff4526fe59 received from {username='slave'} at 127.8.68.200:57893
I20240501 20:10:06.114420 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 8658f3ae7457494faaf046c3c088291d-e97529f5c0244486b7d946ff4526fe59 on tablet e97529f5c0244486b7d946ff4526fe59 with peer 8658f3ae7457494faaf046c3c088291d
I20240501 20:10:06.127748 15249 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: Bootstrap replayed 1/1 log segments. Stats: ops{read=402 overwritten=0 applied=402 ignored=0} inserts{seen=1084 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:06.128087 15249 tablet_bootstrap.cc:492] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: Bootstrap complete.
I20240501 20:10:06.128329 15249 ts_tablet_manager.cc:1367] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: Time spent bootstrapping tablet: real 0.570s	user 0.449s	sys 0.040s
I20240501 20:10:06.128907 15249 raft_consensus.cc:348] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:06.129076 15249 raft_consensus.cc:729] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8658f3ae7457494faaf046c3c088291d, State: Initialized, Role: LEARNER
I20240501 20:10:06.129264 15249 consensus_queue.cc:260] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 402, Last appended: 1.402, Last appended by leader: 402, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:06.129752 15249 ts_tablet_manager.cc:1398] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:10:06.130460 11993 tablet_copy_service.cc:342] P dc71b180f72b4c4697d28b13b363d705: Request end of tablet copy session 8658f3ae7457494faaf046c3c088291d-f8f9c24067b442299db6fa5831feb56d received from {username='slave'} at 127.8.68.200:56713
I20240501 20:10:06.130617 11993 tablet_copy_service.cc:433] P dc71b180f72b4c4697d28b13b363d705: ending tablet copy session 8658f3ae7457494faaf046c3c088291d-f8f9c24067b442299db6fa5831feb56d on tablet f8f9c24067b442299db6fa5831feb56d with peer 8658f3ae7457494faaf046c3c088291d
I20240501 20:10:06.188071 15265 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: Bootstrap replayed 1/1 log segments. Stats: ops{read=379 overwritten=0 applied=379 ignored=0} inserts{seen=1061 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:06.188432 15265 tablet_bootstrap.cc:492] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: Bootstrap complete.
I20240501 20:10:06.188683 15265 ts_tablet_manager.cc:1367] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: Time spent bootstrapping tablet: real 0.504s	user 0.417s	sys 0.042s
I20240501 20:10:06.189234 15265 raft_consensus.cc:348] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:06.189433 15265 raft_consensus.cc:729] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8658f3ae7457494faaf046c3c088291d, State: Initialized, Role: LEARNER
I20240501 20:10:06.189611 15265 consensus_queue.cc:260] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 379, Last appended: 1.379, Last appended by leader: 379, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: true } }
I20240501 20:10:06.190057 15265 ts_tablet_manager.cc:1398] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d: Time spent starting tablet: real 0.001s	user 0.003s	sys 0.000s
I20240501 20:10:06.190733 12123 tablet_copy_service.cc:342] P 34f8af7f2d4245899a7d8fd5012a3721: Request end of tablet copy session 8658f3ae7457494faaf046c3c088291d-7eb828d8511f40c78004d0aaba843df5 received from {username='slave'} at 127.8.68.200:57893
I20240501 20:10:06.190868 12123 tablet_copy_service.cc:433] P 34f8af7f2d4245899a7d8fd5012a3721: ending tablet copy session 8658f3ae7457494faaf046c3c088291d-7eb828d8511f40c78004d0aaba843df5 on tablet 7eb828d8511f40c78004d0aaba843df5 with peer 8658f3ae7457494faaf046c3c088291d
I20240501 20:10:06.216133 15038 raft_consensus.cc:1203] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.348->[1.349-1.349]   Dedup: 1.349->[]
I20240501 20:10:06.507834 15038 raft_consensus.cc:1203] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.401->[1.402-1.402]   Dedup: 1.402->[]
I20240501 20:10:06.551568 15317 raft_consensus.cc:1050] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 8658f3ae7457494faaf046c3c088291d to VOTER
I20240501 20:10:06.552126 15317 consensus_queue.cc:237] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:06.554076 15038 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d [term 2 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:06.554126 13275 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:06.554821 15315 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:06.555168 15316 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:06.559094 15317 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.559841 15038 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.560210 13275 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.564565 11829 catalog_manager.cc:5484] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d reported cstate change: config changed from index 6 to 7, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.566903 15038 raft_consensus.cc:1203] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.378->[1.379-1.379]   Dedup: 1.379->[]
I20240501 20:10:06.571692 12102 consensus_queue.cc:237] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:06.573029 15038 raft_consensus.cc:1261] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d [term 2 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:10:06.573493 15316 consensus_queue.cc:1035] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:06.575732 15316 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.578526 11818 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 262853fe9d5c4aa7acdac809a6378567 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:06.578883 15038 raft_consensus.cc:2932] T 262853fe9d5c4aa7acdac809a6378567 P 8658f3ae7457494faaf046c3c088291d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.579576 11826 catalog_manager.cc:5484] T 262853fe9d5c4aa7acdac809a6378567 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 7 to 8, VOTER 3b95f8c6589e473bb21db7681e227638 (127.8.68.196) evicted. New cstate: current_term: 2 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:06.582787 13257 tablet_service.cc:1514] Processing DeleteTablet for tablet 262853fe9d5c4aa7acdac809a6378567 with delete_type TABLET_DATA_TOMBSTONED (TS 3b95f8c6589e473bb21db7681e227638 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45130
I20240501 20:10:06.583513 15368 tablet_replica.cc:330] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: stopping tablet replica
I20240501 20:10:06.583949 15368 raft_consensus.cc:2229] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:06.584257 15368 raft_consensus.cc:2258] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:06.585855 15368 ts_tablet_manager.cc:1875] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:06.589996 15368 ts_tablet_manager.cc:1888] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 20:10:06.590114 15368 log.cc:1196] T 262853fe9d5c4aa7acdac809a6378567 P 3b95f8c6589e473bb21db7681e227638: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/262853fe9d5c4aa7acdac809a6378567
I20240501 20:10:06.590832 11819 catalog_manager.cc:4830] TS 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351): tablet 262853fe9d5c4aa7acdac809a6378567 (table TestTable [id=6791900190fd42098c6b5729056d1534]) successfully deleted
I20240501 20:10:06.746169 15316 raft_consensus.cc:1050] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 8658f3ae7457494faaf046c3c088291d to VOTER
I20240501 20:10:06.746776 15316 consensus_queue.cc:237] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.349, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:06.748610 12232 raft_consensus.cc:1261] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240501 20:10:06.748652 15038 raft_consensus.cc:1261] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240501 20:10:06.749095 15317 consensus_queue.cc:1035] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240501 20:10:06.749351 15316 consensus_queue.cc:1035] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240501 20:10:06.751854 15316 raft_consensus.cc:2932] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.350: config changed from index 349 to 350, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.752336 12232 raft_consensus.cc:2932] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.756686 11825 catalog_manager.cc:5484] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad reported cstate change: config changed from index 349 to 350, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.766479 15038 raft_consensus.cc:2932] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "18717c9754d04918815985cf07c1ddad" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 42201 } attrs { replace: true } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.767967 12102 consensus_queue.cc:237] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:06.778453 15038 raft_consensus.cc:1261] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240501 20:10:06.779429 15316 consensus_queue.cc:1035] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240501 20:10:06.781426 15317 raft_consensus.cc:2932] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.351: config changed from index 350 to 351, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.781919 15038 raft_consensus.cc:2932] T e97529f5c0244486b7d946ff4526fe59 P 8658f3ae7457494faaf046c3c088291d [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.784746 11818 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e97529f5c0244486b7d946ff4526fe59 with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:06.786180 11826 catalog_manager.cc:5484] T e97529f5c0244486b7d946ff4526fe59 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 350 to 351, VOTER 18717c9754d04918815985cf07c1ddad (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:06.789229 12212 tablet_service.cc:1514] Processing DeleteTablet for tablet e97529f5c0244486b7d946ff4526fe59 with delete_type TABLET_DATA_TOMBSTONED (TS 18717c9754d04918815985cf07c1ddad not found in new config with opid_index 351) from {username='slave'} at 127.0.0.1:50996
I20240501 20:10:06.789852 15377 tablet_replica.cc:330] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad: stopping tablet replica
I20240501 20:10:06.790061 15377 raft_consensus.cc:2229] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:06.790374 15377 raft_consensus.cc:2258] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:06.791989 15377 ts_tablet_manager.cc:1875] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:06.797210 15377 ts_tablet_manager.cc:1888] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.350
I20240501 20:10:06.797322 15377 log.cc:1196] T e97529f5c0244486b7d946ff4526fe59 P 18717c9754d04918815985cf07c1ddad: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/e97529f5c0244486b7d946ff4526fe59
I20240501 20:10:06.798022 11816 catalog_manager.cc:4830] TS 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201): tablet e97529f5c0244486b7d946ff4526fe59 (table rebalance_test_table_1 [id=ba05ecbd1c2c41e293b956c75a0f3457]) successfully deleted
I20240501 20:10:06.954440 15317 raft_consensus.cc:1050] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: attempting to promote NON_VOTER 8658f3ae7457494faaf046c3c088291d to VOTER
I20240501 20:10:06.955199 15317 consensus_queue.cc:237] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 379, Committed index: 379, Last appended: 1.379, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:06.957178 15038 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 379. Preceding OpId from leader: term: 1 index: 380. (index mismatch)
I20240501 20:10:06.957248 13275 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Refusing update from remote peer 34f8af7f2d4245899a7d8fd5012a3721: Log matching property violated. Preceding OpId in replica: term: 1 index: 379. Preceding OpId from leader: term: 1 index: 380. (index mismatch)
I20240501 20:10:06.957739 15315 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 379, Time since last communication: 0.000s
I20240501 20:10:06.958029 15316 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 379, Time since last communication: 0.000s
I20240501 20:10:06.960729 15315 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Committing config change with OpId 1.380: config changed from index 379 to 380, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.965066 13275 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Committing config change with OpId 1.380: config changed from index 379 to 380, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.971051 11825 catalog_manager.cc:5484] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 reported cstate change: config changed from index 379 to 380, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34f8af7f2d4245899a7d8fd5012a3721" committed_config { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:06.971742 15038 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 1 FOLLOWER]: Committing config change with OpId 1.380: config changed from index 379 to 380, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.986356 15339 raft_consensus.cc:1050] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: attempting to promote NON_VOTER 8658f3ae7457494faaf046c3c088291d to VOTER
I20240501 20:10:06.986948 15339 consensus_queue.cc:237] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 402, Committed index: 402, Last appended: 1.402, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:06.988968 13667 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 402. Preceding OpId from leader: term: 1 index: 403. (index mismatch)
I20240501 20:10:06.989004 15038 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 1 LEARNER]: Refusing update from remote peer dc71b180f72b4c4697d28b13b363d705: Log matching property violated. Preceding OpId in replica: term: 1 index: 402. Preceding OpId from leader: term: 1 index: 403. (index mismatch)
I20240501 20:10:06.989534 14868 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 403, Last known committed idx: 402, Time since last communication: 0.000s
I20240501 20:10:06.989833 15339 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 403, Last known committed idx: 402, Time since last communication: 0.000s
I20240501 20:10:06.992202 14868 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Committing config change with OpId 1.403: config changed from index 402 to 403, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.992798 13667 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Committing config change with OpId 1.403: config changed from index 402 to 403, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.993609 15037 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 1 FOLLOWER]: Committing config change with OpId 1.403: config changed from index 402 to 403, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:06.997577 11826 catalog_manager.cc:5484] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 reported cstate change: config changed from index 402 to 403, 8658f3ae7457494faaf046c3c088291d (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dc71b180f72b4c4697d28b13b363d705" committed_config { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:07.051642 11972 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8f9c24067b442299db6fa5831feb56d"
dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51450
I20240501 20:10:07.051828 11972 raft_consensus.cc:595] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:07.067260 12102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7eb828d8511f40c78004d0aaba843df5"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56660
I20240501 20:10:07.067440 12102 raft_consensus.cc:595] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:07.283299 11972 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8f9c24067b442299db6fa5831feb56d"
dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51462
I20240501 20:10:07.283495 11972 raft_consensus.cc:595] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:07.298472 12102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7eb828d8511f40c78004d0aaba843df5"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56678
I20240501 20:10:07.298674 12102 raft_consensus.cc:595] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:07.451072 15339 raft_consensus.cc:979] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: : Instructing follower aed522b44a8e420daf65718450541c3c to start an election
I20240501 20:10:07.451238 15339 raft_consensus.cc:1067] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Signalling peer aed522b44a8e420daf65718450541c3c to start an election
I20240501 20:10:07.451841 13667 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8f9c24067b442299db6fa5831feb56d"
dest_uuid: "aed522b44a8e420daf65718450541c3c"
 from {username='slave'} at 127.8.68.193:56951
I20240501 20:10:07.452090 13667 raft_consensus.cc:482] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:07.452185 13667 raft_consensus.cc:3037] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:07.454039 13667 raft_consensus.cc:504] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:07.455399 13667 leader_election.cc:288] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [CANDIDATE]: Term 2 election: Requested vote from peers dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397), 8658f3ae7457494faaf046c3c088291d (127.8.68.200:40535)
I20240501 20:10:07.455899 11972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f9c24067b442299db6fa5831feb56d" candidate_uuid: "aed522b44a8e420daf65718450541c3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 403 } } ignore_live_leader: true dest_uuid: "dc71b180f72b4c4697d28b13b363d705"
I20240501 20:10:07.456102 11972 raft_consensus.cc:3032] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:07.456179 11972 raft_consensus.cc:729] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dc71b180f72b4c4697d28b13b363d705, State: Running, Role: LEADER
I20240501 20:10:07.456423 11972 consensus_queue.cc:260] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 403, Committed index: 403, Last appended: 1.403, Last appended by leader: 403, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:07.456763 11972 raft_consensus.cc:3037] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:07.458600 11972 raft_consensus.cc:2454] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aed522b44a8e420daf65718450541c3c in term 2.
I20240501 20:10:07.459054 13602 leader_election.cc:302] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aed522b44a8e420daf65718450541c3c, dc71b180f72b4c4697d28b13b363d705; no voters: 
I20240501 20:10:07.459451 15409 raft_consensus.cc:2781] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:07.459906 15409 raft_consensus.cc:686] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 2 LEADER]: Becoming Leader. State: Replica: aed522b44a8e420daf65718450541c3c, State: Running, Role: LEADER
I20240501 20:10:07.460427 15409 consensus_queue.cc:237] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 403, Committed index: 403, Last appended: 1.403, Last appended by leader: 403, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:07.462584 15038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f9c24067b442299db6fa5831feb56d" candidate_uuid: "aed522b44a8e420daf65718450541c3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 403 } } ignore_live_leader: true dest_uuid: "8658f3ae7457494faaf046c3c088291d"
I20240501 20:10:07.462867 15038 raft_consensus.cc:3037] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:07.463748 11829 catalog_manager.cc:5484] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c reported cstate change: term changed from 1 to 2, leader changed from dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) to aed522b44a8e420daf65718450541c3c (127.8.68.199). New cstate: current_term: 2 leader_uuid: "aed522b44a8e420daf65718450541c3c" committed_config { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:07.464929 15038 raft_consensus.cc:2454] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aed522b44a8e420daf65718450541c3c in term 2.
I20240501 20:10:07.518388 15316 raft_consensus.cc:979] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: : Instructing follower 8658f3ae7457494faaf046c3c088291d to start an election
I20240501 20:10:07.518543 15470 raft_consensus.cc:1067] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Signalling peer 8658f3ae7457494faaf046c3c088291d to start an election
I20240501 20:10:07.519263 15038 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7eb828d8511f40c78004d0aaba843df5"
dest_uuid: "8658f3ae7457494faaf046c3c088291d"
 from {username='slave'} at 127.8.68.194:51467
I20240501 20:10:07.519505 15038 raft_consensus.cc:482] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:07.519630 15038 raft_consensus.cc:3037] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:07.522148 15038 raft_consensus.cc:504] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:07.522878 12102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7eb828d8511f40c78004d0aaba843df5"
dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56696
I20240501 20:10:07.523097 12102 raft_consensus.cc:595] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:07.523864 15038 leader_election.cc:288] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [CANDIDATE]: Term 2 election: Requested vote from peers 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853), 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351)
I20240501 20:10:07.524691 12102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb828d8511f40c78004d0aaba843df5" candidate_uuid: "8658f3ae7457494faaf046c3c088291d" candidate_term: 2 candidate_status { last_received { term: 1 index: 380 } } ignore_live_leader: true dest_uuid: "34f8af7f2d4245899a7d8fd5012a3721"
I20240501 20:10:07.524890 12102 raft_consensus.cc:3032] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:07.524986 12102 raft_consensus.cc:729] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 34f8af7f2d4245899a7d8fd5012a3721, State: Running, Role: LEADER
I20240501 20:10:07.525259 12102 consensus_queue.cc:260] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 1.380, Last appended by leader: 380, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:07.525557 12102 raft_consensus.cc:3037] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:07.527422 12102 raft_consensus.cc:2454] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8658f3ae7457494faaf046c3c088291d in term 2.
I20240501 20:10:07.527937 14973 leader_election.cc:302] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34f8af7f2d4245899a7d8fd5012a3721, 8658f3ae7457494faaf046c3c088291d; no voters: 
I20240501 20:10:07.528643 15473 raft_consensus.cc:2781] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:07.529083 15473 raft_consensus.cc:686] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 2 LEADER]: Becoming Leader. State: Replica: 8658f3ae7457494faaf046c3c088291d, State: Running, Role: LEADER
I20240501 20:10:07.529628 15473 consensus_queue.cc:237] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 1.380, Last appended by leader: 380, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:07.533215 11829 catalog_manager.cc:5484] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d reported cstate change: term changed from 1 to 2, leader changed from 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) to 8658f3ae7457494faaf046c3c088291d (127.8.68.200). New cstate: current_term: 2 leader_uuid: "8658f3ae7457494faaf046c3c088291d" committed_config { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:07.535580 13275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb828d8511f40c78004d0aaba843df5" candidate_uuid: "8658f3ae7457494faaf046c3c088291d" candidate_term: 2 candidate_status { last_received { term: 1 index: 380 } } ignore_live_leader: true dest_uuid: "3b95f8c6589e473bb21db7681e227638"
I20240501 20:10:07.535820 13275 raft_consensus.cc:3037] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:07.538672 13275 raft_consensus.cc:2454] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8658f3ae7457494faaf046c3c088291d in term 2.
I20240501 20:10:07.970059 11972 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Refusing update from remote peer aed522b44a8e420daf65718450541c3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 403. Preceding OpId from leader: term: 2 index: 404. (index mismatch)
I20240501 20:10:07.970654 15496 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc71b180f72b4c4697d28b13b363d705" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 404, Last known committed idx: 403, Time since last communication: 0.000s
I20240501 20:10:07.974208 15038 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 2 FOLLOWER]: Refusing update from remote peer aed522b44a8e420daf65718450541c3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 403. Preceding OpId from leader: term: 2 index: 404. (index mismatch)
I20240501 20:10:07.974869 15496 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 404, Last known committed idx: 403, Time since last communication: 0.000s
I20240501 20:10:07.979142 12102 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Refusing update from remote peer 8658f3ae7457494faaf046c3c088291d: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 2 index: 381. (index mismatch)
I20240501 20:10:07.979758 15473 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f8af7f2d4245899a7d8fd5012a3721" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 32853 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I20240501 20:10:07.983309 13275 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer 8658f3ae7457494faaf046c3c088291d: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 2 index: 381. (index mismatch)
I20240501 20:10:07.983875 15473 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I20240501 20:10:08.418221 15038 consensus_queue.cc:237] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 381, Committed index: 381, Last appended: 2.381, Last appended by leader: 380, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:08.419728 13275 raft_consensus.cc:1261] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Refusing update from remote peer 8658f3ae7457494faaf046c3c088291d: Log matching property violated. Preceding OpId in replica: term: 2 index: 381. Preceding OpId from leader: term: 2 index: 382. (index mismatch)
I20240501 20:10:08.420183 15527 consensus_queue.cc:1035] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 382, Last known committed idx: 381, Time since last communication: 0.000s
I20240501 20:10:08.422252 15474 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d [term 2 LEADER]: Committing config change with OpId 2.382: config changed from index 380 to 382, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:08.422827 13275 raft_consensus.cc:2932] T 7eb828d8511f40c78004d0aaba843df5 P 3b95f8c6589e473bb21db7681e227638 [term 2 FOLLOWER]: Committing config change with OpId 2.382: config changed from index 380 to 382, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:08.425173 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7eb828d8511f40c78004d0aaba843df5 with cas_config_opid_index 380: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.426674 11829 catalog_manager.cc:5484] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d reported cstate change: config changed from index 380 to 382, VOTER 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "8658f3ae7457494faaf046c3c088291d" committed_config { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "3b95f8c6589e473bb21db7681e227638" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.430676 12083 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb828d8511f40c78004d0aaba843df5 with delete_type TABLET_DATA_TOMBSTONED (TS 34f8af7f2d4245899a7d8fd5012a3721 not found in new config with opid_index 382) from {username='slave'} at 127.0.0.1:42804
I20240501 20:10:08.431195 15532 tablet_replica.cc:330] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: stopping tablet replica
I20240501 20:10:08.431344 15532 raft_consensus.cc:2229] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.431489 15532 raft_consensus.cc:2258] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.432699 15532 ts_tablet_manager.cc:1875] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.436981 15532 ts_tablet_manager.cc:1888] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.381
I20240501 20:10:08.437075 15532 log.cc:1196] T 7eb828d8511f40c78004d0aaba843df5 P 34f8af7f2d4245899a7d8fd5012a3721: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/7eb828d8511f40c78004d0aaba843df5
I20240501 20:10:08.437763 11818 catalog_manager.cc:4830] TS 34f8af7f2d4245899a7d8fd5012a3721 (127.8.68.194:32853): tablet 7eb828d8511f40c78004d0aaba843df5 (table rebalance_test_table_3 [id=0e909cfb82614206afe575f03aece87c]) successfully deleted
I20240501 20:10:08.493697 13667 consensus_queue.cc:237] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 2.404, Last appended by leader: 403, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } }
I20240501 20:10:08.495278 15038 raft_consensus.cc:1261] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 2 FOLLOWER]: Refusing update from remote peer aed522b44a8e420daf65718450541c3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 404. Preceding OpId from leader: term: 2 index: 405. (index mismatch)
I20240501 20:10:08.495703 15535 consensus_queue.cc:1035] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I20240501 20:10:08.497592 15534 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c [term 2 LEADER]: Committing config change with OpId 2.405: config changed from index 403 to 405, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:08.498183 15038 raft_consensus.cc:2932] T f8f9c24067b442299db6fa5831feb56d P 8658f3ae7457494faaf046c3c088291d [term 2 FOLLOWER]: Committing config change with OpId 2.405: config changed from index 403 to 405, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } } }
I20240501 20:10:08.500293 11816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8f9c24067b442299db6fa5831feb56d with cas_config_opid_index 403: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.501513 11825 catalog_manager.cc:5484] T f8f9c24067b442299db6fa5831feb56d P aed522b44a8e420daf65718450541c3c reported cstate change: config changed from index 403 to 405, VOTER dc71b180f72b4c4697d28b13b363d705 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "aed522b44a8e420daf65718450541c3c" committed_config { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "aed522b44a8e420daf65718450541c3c" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 40105 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8658f3ae7457494faaf046c3c088291d" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 40535 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.504729 11952 tablet_service.cc:1514] Processing DeleteTablet for tablet f8f9c24067b442299db6fa5831feb56d with delete_type TABLET_DATA_TOMBSTONED (TS dc71b180f72b4c4697d28b13b363d705 not found in new config with opid_index 405) from {username='slave'} at 127.0.0.1:58526
I20240501 20:10:08.505374 15536 tablet_replica.cc:330] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: stopping tablet replica
I20240501 20:10:08.505540 15536 raft_consensus.cc:2229] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.505718 15536 raft_consensus.cc:2258] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.507233 15536 ts_tablet_manager.cc:1875] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.511227 15536 ts_tablet_manager.cc:1888] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.404
I20240501 20:10:08.511332 15536 log.cc:1196] T f8f9c24067b442299db6fa5831feb56d P dc71b180f72b4c4697d28b13b363d705: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/f8f9c24067b442299db6fa5831feb56d
I20240501 20:10:08.511988 11816 catalog_manager.cc:4830] TS dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397): tablet f8f9c24067b442299db6fa5831feb56d (table rebalance_test_table_4 [id=414b19c8cda24a83b02c93fd85e71b3a]) successfully deleted
I20240501 20:10:08.704032 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:08.722048 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:08.723930 12083 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:08.766899 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:08.783068 13649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:08.783161 15018 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:08.806821 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:08.828902 11952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:08.976979 12083 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:08.988590 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:08.994508 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:08.995043 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:09.006455 13649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:09.006476 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:09.007195 15018 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:09.025715 11952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:09.330529 12083 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:09.332557 12212 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:09.339918 13257 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:09.357733 15018 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:09.369809 13388 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:09.377002 13649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:09.390627 13519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:09.423629 11952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 466d4986891e4af2a4e44757733bfd5c | 127.8.68.254:46165 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.8.68.212:43765 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 18717c9754d04918815985cf07c1ddad | 127.8.68.195:42201 | HEALTHY | <none>   |       3        |       0
 34f8af7f2d4245899a7d8fd5012a3721 | 127.8.68.194:32853 | HEALTHY | <none>   |       4        |       0
 3b95f8c6589e473bb21db7681e227638 | 127.8.68.196:36351 | HEALTHY | <none>   |       1        |       0
 463bd46a1f6e4e89a3040c4cf50dd919 | 127.8.68.198:35531 | HEALTHY | <none>   |       2        |       0
 535367472879403fa999dd00a155f042 | 127.8.68.197:43403 | HEALTHY | <none>   |       2        |       0
 8658f3ae7457494faaf046c3c088291d | 127.8.68.200:40535 | HEALTHY | <none>   |       1        |       0
 aed522b44a8e420daf65718450541c3c | 127.8.68.199:40105 | HEALTHY | <none>   |       1        |       0
 dc71b180f72b4c4697d28b13b363d705 | 127.8.68.193:39397 | HEALTHY | <none>   |       2        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.8.68.193                                                                                                                                                    | experimental          | 127.8.68.193:39397
 local_ip_for_outbound_sockets            | 127.8.68.194                                                                                                                                                    | experimental          | 127.8.68.194:32853
 local_ip_for_outbound_sockets            | 127.8.68.195                                                                                                                                                    | experimental          | 127.8.68.195:42201
 local_ip_for_outbound_sockets            | 127.8.68.196                                                                                                                                                    | experimental          | 127.8.68.196:36351
 local_ip_for_outbound_sockets            | 127.8.68.197                                                                                                                                                    | experimental          | 127.8.68.197:43403
 local_ip_for_outbound_sockets            | 127.8.68.198                                                                                                                                                    | experimental          | 127.8.68.198:35531
 local_ip_for_outbound_sockets            | 127.8.68.199                                                                                                                                                    | experimental          | 127.8.68.199:40105
 local_ip_for_outbound_sockets            | 127.8.68.200                                                                                                                                                    | experimental          | 127.8.68.200:40535
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                            | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.8.68.193:39397
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.8.68.194:32853
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.8.68.195:42201
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.8.68.196:36351
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.8.68.197:43403
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.8.68.198:35531
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.8.68.199:40105
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.8.68.200:40535

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.8.68.212:43765 | all 8 server(s) checked
 time_source         | builtin            | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240501 20:10:09.485699  8467 log_verifier.cc:126] Checking tablet 0faf9b33582449ee9ffa353ae6ea3d6a
I20240501 20:10:09.636375  8467 log_verifier.cc:177] Verified matching terms for 435 ops in tablet 0faf9b33582449ee9ffa353ae6ea3d6a
I20240501 20:10:09.636884  8467 log_verifier.cc:126] Checking tablet 262853fe9d5c4aa7acdac809a6378567
I20240501 20:10:09.650631  8467 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 262853fe9d5c4aa7acdac809a6378567
I20240501 20:10:09.650713  8467 log_verifier.cc:126] Checking tablet 2c0a1865e31241569216a3b4b072829c
I20240501 20:10:09.819164  8467 log_verifier.cc:177] Verified matching terms for 509 ops in tablet 2c0a1865e31241569216a3b4b072829c
I20240501 20:10:09.819751  8467 log_verifier.cc:126] Checking tablet 321fb505be2f4263a79c9bcba4be6616
I20240501 20:10:09.961516  8467 log_verifier.cc:177] Verified matching terms for 376 ops in tablet 321fb505be2f4263a79c9bcba4be6616
I20240501 20:10:09.961975  8467 log_verifier.cc:126] Checking tablet 38b8e2865d134700ad0959372afc6670
I20240501 20:10:10.111075  8467 log_verifier.cc:177] Verified matching terms for 434 ops in tablet 38b8e2865d134700ad0959372afc6670
I20240501 20:10:10.111620  8467 log_verifier.cc:126] Checking tablet 4ad26a211a9c4c609da559d2681c837a
I20240501 20:10:10.268234  8467 log_verifier.cc:177] Verified matching terms for 444 ops in tablet 4ad26a211a9c4c609da559d2681c837a
I20240501 20:10:10.268719  8467 log_verifier.cc:126] Checking tablet 575f6e5055d84dc38ef2326cec06fa47
I20240501 20:10:10.412814  8467 log_verifier.cc:177] Verified matching terms for 396 ops in tablet 575f6e5055d84dc38ef2326cec06fa47
I20240501 20:10:10.413565  8467 log_verifier.cc:126] Checking tablet 67043e58e5fd4233bc945432eeae1227
I20240501 20:10:10.572494  8467 log_verifier.cc:177] Verified matching terms for 450 ops in tablet 67043e58e5fd4233bc945432eeae1227
I20240501 20:10:10.573000  8467 log_verifier.cc:126] Checking tablet 7eb828d8511f40c78004d0aaba843df5
I20240501 20:10:10.704334  8467 log_verifier.cc:177] Verified matching terms for 382 ops in tablet 7eb828d8511f40c78004d0aaba843df5
I20240501 20:10:10.704825  8467 log_verifier.cc:126] Checking tablet 8baf6913f38a42c3a369cd673ff9438d
I20240501 20:10:10.842473  8467 log_verifier.cc:177] Verified matching terms for 416 ops in tablet 8baf6913f38a42c3a369cd673ff9438d
I20240501 20:10:10.842969  8467 log_verifier.cc:126] Checking tablet 9d7dd13822784e789d558b4cc6c5010e
I20240501 20:10:10.971985  8467 log_verifier.cc:177] Verified matching terms for 375 ops in tablet 9d7dd13822784e789d558b4cc6c5010e
I20240501 20:10:10.972437  8467 log_verifier.cc:126] Checking tablet cf6c676aa51e44a9b8a1c2a3c828ccd0
I20240501 20:10:11.131507  8467 log_verifier.cc:177] Verified matching terms for 447 ops in tablet cf6c676aa51e44a9b8a1c2a3c828ccd0
I20240501 20:10:11.131970  8467 log_verifier.cc:126] Checking tablet d847215537454b4ba9e8a8b87ec91699
I20240501 20:10:11.293097  8467 log_verifier.cc:177] Verified matching terms for 466 ops in tablet d847215537454b4ba9e8a8b87ec91699
I20240501 20:10:11.293888  8467 log_verifier.cc:126] Checking tablet e97529f5c0244486b7d946ff4526fe59
I20240501 20:10:11.419624  8467 log_verifier.cc:177] Verified matching terms for 351 ops in tablet e97529f5c0244486b7d946ff4526fe59
I20240501 20:10:11.420063  8467 log_verifier.cc:126] Checking tablet f25dcb94e89c4c42979fdb39523b7fd2
I20240501 20:10:11.562179  8467 log_verifier.cc:177] Verified matching terms for 408 ops in tablet f25dcb94e89c4c42979fdb39523b7fd2
I20240501 20:10:11.562695  8467 log_verifier.cc:126] Checking tablet f8f9c24067b442299db6fa5831feb56d
I20240501 20:10:11.727938  8467 log_verifier.cc:177] Verified matching terms for 405 ops in tablet f8f9c24067b442299db6fa5831feb56d
I20240501 20:10:11.735095  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 11891
I20240501 20:10:11.736155 12013 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:11.963686  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 11891
W20240501 20:10:12.000481 13604 connection.cc:535] server connection from 127.8.68.193:56951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:12.001227  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 12021
I20240501 20:10:12.002247 12143 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:12.045017 13341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:39397: connect: Connection refused (error 111)
W20240501 20:10:12.046620 13341 consensus_peers.cc:485] T 67043e58e5fd4233bc945432eeae1227 P 535367472879403fa999dd00a155f042 -> Peer dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397): Couldn't send request to peer dc71b180f72b4c4697d28b13b363d705. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:39397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:12.189931 12166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:39397: connect: Connection refused (error 111)
W20240501 20:10:12.191254 12166 consensus_peers.cc:485] T 321fb505be2f4263a79c9bcba4be6616 P 18717c9754d04918815985cf07c1ddad -> Peer dc71b180f72b4c4697d28b13b363d705 (127.8.68.193:39397): Couldn't send request to peer dc71b180f72b4c4697d28b13b363d705. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:39397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:12.237032  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 12021
W20240501 20:10:12.272917 14972 connection.cc:535] server connection from 127.8.68.194:51467 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:12.273299  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 12151
I20240501 20:10:12.274181 12272 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:12.508692  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 12151
W20240501 20:10:12.546655 13471 connection.cc:535] server connection from 127.8.68.195:41551 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:12.546996 13471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.68.195:42201 (error 108)
I20240501 20:10:12.547220  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 13195
W20240501 20:10:12.549109 13471 consensus_peers.cc:485] T 38b8e2865d134700ad0959372afc6670 P 463bd46a1f6e4e89a3040c4cf50dd919 -> Peer 18717c9754d04918815985cf07c1ddad (127.8.68.195:42201): Couldn't send request to peer 18717c9754d04918815985cf07c1ddad. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.195:42201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:12.553210 13317 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:12.717002  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 13195
I20240501 20:10:12.743222  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 13326
I20240501 20:10:12.748800 13447 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:12.771865 14974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.196:36351: connect: Connection refused (error 111)
W20240501 20:10:12.773231 14974 consensus_peers.cc:485] T 7eb828d8511f40c78004d0aaba843df5 P 8658f3ae7457494faaf046c3c088291d -> Peer 3b95f8c6589e473bb21db7681e227638 (127.8.68.196:36351): Couldn't send request to peer 3b95f8c6589e473bb21db7681e227638. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.196:36351: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:12.920099  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 13326
I20240501 20:10:12.945644  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 13456
I20240501 20:10:12.950922 13578 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:13.111040  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 13456
I20240501 20:10:13.135797  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 13587
I20240501 20:10:13.141872 13709 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:13.304333 15692 raft_consensus.cc:482] T 2c0a1865e31241569216a3b4b072829c P aed522b44a8e420daf65718450541c3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc71b180f72b4c4697d28b13b363d705)
I20240501 20:10:13.304685  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 13587
I20240501 20:10:13.329661  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 14955
I20240501 20:10:13.335733 15078 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:13.458222  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 14955
I20240501 20:10:13.475471  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 11801
I20240501 20:10:13.476430 11861 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:13.632810  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 11801
2024-05-01T20:10:13Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/3 (47270 ms)
[----------] 1 test from RebalanceParamTest (47271 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240501 20:10:13.707360  8467 test_util.cc:268] Using random seed: 319471188
I20240501 20:10:13.708900  8467 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:10:13.708951  8467 ts_itest-base.cc:113] --------------
I20240501 20:10:13.708971  8467 ts_itest-base.cc:114] 8 tablet servers
I20240501 20:10:13.709004  8467 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:10:13.709045  8467 ts_itest-base.cc:116] --------------
2024-05-01T20:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:10:13Z Disabled control of system clock
I20240501 20:10:13.754384  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:46529
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:46529
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:10:14.064909 15703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:14.065169 15703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:14.065287 15703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:14.072324 15703 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:14.072423 15703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:10:14.072497 15703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:14.072571 15703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:10:14.072631 15703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:10:14.081097 15703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:46529
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:14.082155 15703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:14.083751 15703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:14.093667 15708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:14.094049 15709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:14.094527 15711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:14.095019 15703 server_base.cc:1034] running on GCE node
I20240501 20:10:14.095621 15703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:14.096838 15703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:14.098052 15703 hybrid_clock.cc:648] HybridClock initialized: now 1714594214097988 us; error 34 us; skew 500 ppm
I20240501 20:10:14.098397 15703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:14.100406 15703 webserver.cc:460] Webserver started at http://127.8.68.254:38751/ using document root <none> and password file <none>
I20240501 20:10:14.101063 15703 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:14.101167 15703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:14.101475 15703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:14.103906 15703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9601eb27ae55460e9ce7240536578d63"
format_stamp: "Formatted at 2024-05-01 20:10:14 on dist-test-slave-08p9"
I20240501 20:10:14.104614 15703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9601eb27ae55460e9ce7240536578d63"
format_stamp: "Formatted at 2024-05-01 20:10:14 on dist-test-slave-08p9"
I20240501 20:10:14.109285 15703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240501 20:10:14.112290 15717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:14.113483 15703 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:10:14.113644 15703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9601eb27ae55460e9ce7240536578d63"
format_stamp: "Formatted at 2024-05-01 20:10:14 on dist-test-slave-08p9"
I20240501 20:10:14.113831 15703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:14.140950 15703 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:14.141642 15703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:14.141896 15703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:14.158011 15703 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:46529
I20240501 20:10:14.158030 15768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:46529 every 8 connection(s)
I20240501 20:10:14.159207 15703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:10:14.162382 15769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.165897  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 15703
I20240501 20:10:14.166112  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:10:14.171938 15769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63: Bootstrap starting.
I20240501 20:10:14.175051 15769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.176259 15769 log.cc:824] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:14.178859 15769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63: No bootstrap required, opened a new log
I20240501 20:10:14.184907 15769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9601eb27ae55460e9ce7240536578d63" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46529 } }
I20240501 20:10:14.185200 15769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.185273 15769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9601eb27ae55460e9ce7240536578d63, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.185740 15769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9601eb27ae55460e9ce7240536578d63" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46529 } }
I20240501 20:10:14.185912 15769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:10:14.186002 15769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:10:14.186120 15769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.187987 15769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9601eb27ae55460e9ce7240536578d63" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46529 } }
I20240501 20:10:14.188396 15769 leader_election.cc:302] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9601eb27ae55460e9ce7240536578d63; no voters: 
I20240501 20:10:14.188865 15769 leader_election.cc:288] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:10:14.188975 15774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:14.189541 15774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [term 1 LEADER]: Becoming Leader. State: Replica: 9601eb27ae55460e9ce7240536578d63, State: Running, Role: LEADER
I20240501 20:10:14.189988 15774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9601eb27ae55460e9ce7240536578d63" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46529 } }
I20240501 20:10:14.190579 15769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:10:14.192585 15776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9601eb27ae55460e9ce7240536578d63. Latest consensus state: current_term: 1 leader_uuid: "9601eb27ae55460e9ce7240536578d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9601eb27ae55460e9ce7240536578d63" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46529 } } }
I20240501 20:10:14.192644 15775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9601eb27ae55460e9ce7240536578d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9601eb27ae55460e9ce7240536578d63" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46529 } } }
I20240501 20:10:14.192893 15776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [sys.catalog]: This master's current role is: LEADER
I20240501 20:10:14.192941 15775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63 [sys.catalog]: This master's current role is: LEADER
I20240501 20:10:14.194427 15779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:10:14.199518 15779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:10:14.206914 15779 catalog_manager.cc:1347] Generated new cluster ID: 5e141a8fa23e4da0bfac36c594e475ca
I20240501 20:10:14.207010 15779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:10:14.241360 15779 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:10:14.242439 15779 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:10:14.264410 15779 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9601eb27ae55460e9ce7240536578d63: Generated new TSK 0
I20240501 20:10:14.265079 15779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:10:14.279410  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:10:14.596654 15793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:14.596887 15793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:14.597013 15793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:14.604166 15793 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:14.604265 15793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:14.604408 15793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240501 20:10:14.612890 15793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:14.613950 15793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:14.615553 15793 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:14.626626 15799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:14.627100 15801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:14.627601 15798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:14.628145 15793 server_base.cc:1034] running on GCE node
I20240501 20:10:14.628620 15793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:14.629395 15793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:14.630578 15793 hybrid_clock.cc:648] HybridClock initialized: now 1714594214630533 us; error 47 us; skew 500 ppm
I20240501 20:10:14.630858 15793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:14.632844 15793 webserver.cc:460] Webserver started at http://127.8.68.193:46663/ using document root <none> and password file <none>
I20240501 20:10:14.633479 15793 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:14.633569 15793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:14.633888 15793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:14.636276 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
format_stamp: "Formatted at 2024-05-01 20:10:14 on dist-test-slave-08p9"
I20240501 20:10:14.636971 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
format_stamp: "Formatted at 2024-05-01 20:10:14 on dist-test-slave-08p9"
I20240501 20:10:14.641603 15793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20240501 20:10:14.644636 15807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:14.645756 15793 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:14.645893 15793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
format_stamp: "Formatted at 2024-05-01 20:10:14 on dist-test-slave-08p9"
I20240501 20:10:14.646060 15793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:14.682479 15793 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:14.683269 15793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:14.683516 15793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:14.684422 15793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:14.685810 15793 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:14.685901 15793 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:14.686003 15793 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:14.686076 15793 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:14.714521 15793 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:35513
I20240501 20:10:14.714591 15919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:35513 every 8 connection(s)
I20240501 20:10:14.715729 15793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:10:14.720602  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 15793
I20240501 20:10:14.720786  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:10:14.724418  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:14.728852 15920 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:14.729108 15920 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:14.729828 15920 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:14.731621 15734 ts_manager.cc:194] Registered new tserver with Master: 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:14.733047 15734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:51397
W20240501 20:10:15.031514 15924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:15.031749 15924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:15.031839 15924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:15.038800 15924 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:15.038893 15924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:15.039000 15924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240501 20:10:15.047410 15924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:15.048416 15924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:15.049888 15924 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:15.060679 15929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:15.061156 15930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:15.061537 15932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:15.062165 15924 server_base.cc:1034] running on GCE node
I20240501 20:10:15.062675 15924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:15.063462 15924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:15.064651 15924 hybrid_clock.cc:648] HybridClock initialized: now 1714594215064619 us; error 31 us; skew 500 ppm
I20240501 20:10:15.064927 15924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:15.066875 15924 webserver.cc:460] Webserver started at http://127.8.68.194:41251/ using document root <none> and password file <none>
I20240501 20:10:15.067508 15924 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:15.067610 15924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:15.067895 15924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:15.071800 15924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "754fcd3b82f041568f546c6205f44272"
format_stamp: "Formatted at 2024-05-01 20:10:15 on dist-test-slave-08p9"
I20240501 20:10:15.072499 15924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "754fcd3b82f041568f546c6205f44272"
format_stamp: "Formatted at 2024-05-01 20:10:15 on dist-test-slave-08p9"
I20240501 20:10:15.076951 15924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240501 20:10:15.079916 15938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:15.081028 15924 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:15.081168 15924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "754fcd3b82f041568f546c6205f44272"
format_stamp: "Formatted at 2024-05-01 20:10:15 on dist-test-slave-08p9"
I20240501 20:10:15.081334 15924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:15.100338 15924 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:15.100992 15924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:15.101236 15924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:15.102047 15924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:15.103374 15924 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:15.103466 15924 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:15.103565 15924 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:15.103636 15924 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:15.131887 15924 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:46677
I20240501 20:10:15.131953 16050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:46677 every 8 connection(s)
I20240501 20:10:15.133006 15924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:10:15.141896  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 15924
I20240501 20:10:15.142066  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:10:15.143901 16051 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:15.144171 16051 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:15.144948 16051 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:15.145063  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:15.146126 15734 ts_manager.cc:194] Registered new tserver with Master: 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:15.146872 15734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:41263
W20240501 20:10:15.461010 16054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:15.461372 16054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:15.461496 16054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:15.469632 16054 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:15.469753 16054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:15.469888 16054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240501 20:10:15.479084 16054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:15.480350 16054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:15.482121 16054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:15.493690 16060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:15.494364 16059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:15.494724 16062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:15.495285 16054 server_base.cc:1034] running on GCE node
I20240501 20:10:15.495870 16054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:15.496735 16054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:15.497953 16054 hybrid_clock.cc:648] HybridClock initialized: now 1714594215497904 us; error 39 us; skew 500 ppm
I20240501 20:10:15.498301 16054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:15.504828 16054 webserver.cc:460] Webserver started at http://127.8.68.195:44697/ using document root <none> and password file <none>
I20240501 20:10:15.505555 16054 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:15.505682 16054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:15.506018 16054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:15.508755 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "042aee20cf4e4e63be3ecc013fda59c9"
format_stamp: "Formatted at 2024-05-01 20:10:15 on dist-test-slave-08p9"
I20240501 20:10:15.509538 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "042aee20cf4e4e63be3ecc013fda59c9"
format_stamp: "Formatted at 2024-05-01 20:10:15 on dist-test-slave-08p9"
I20240501 20:10:15.514602 16054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.007s
I20240501 20:10:15.517725 16068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:15.519073 16054 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 20:10:15.519222 16054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "042aee20cf4e4e63be3ecc013fda59c9"
format_stamp: "Formatted at 2024-05-01 20:10:15 on dist-test-slave-08p9"
I20240501 20:10:15.519399 16054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:15.534271 16054 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:15.534997 16054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:15.535245 16054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:15.536079 16054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:15.537389 16054 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:15.537480 16054 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:15.537581 16054 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:15.537653 16054 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:15.565852 16054 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:46553
I20240501 20:10:15.565909 16180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:46553 every 8 connection(s)
I20240501 20:10:15.567047 16054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:10:15.573045  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 16054
I20240501 20:10:15.573251  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:10:15.576943  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:15.579697 16181 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:15.580036 16181 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:15.581104 16181 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:15.582773 15734 ts_manager.cc:194] Registered new tserver with Master: 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:15.583745 15734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:41061
I20240501 20:10:15.735566 15920 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
W20240501 20:10:15.909897 16184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:15.910154 16184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:15.910301 16184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:15.917225 16184 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:15.917335 16184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:15.917479 16184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:10:15.925850 16184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:15.926982 16184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:15.928529 16184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:15.939054 16189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:15.939692 16192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:15.940068 16190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:15.940626 16184 server_base.cc:1034] running on GCE node
I20240501 20:10:15.941156 16184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:15.941951 16184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:15.943205 16184 hybrid_clock.cc:648] HybridClock initialized: now 1714594215943125 us; error 54 us; skew 500 ppm
I20240501 20:10:15.943497 16184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:15.945449 16184 webserver.cc:460] Webserver started at http://127.8.68.196:40387/ using document root <none> and password file <none>
I20240501 20:10:15.946084 16184 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:15.946187 16184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:15.946511 16184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:15.948726 16184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f856e0d1074b431eb19763847958b89e"
format_stamp: "Formatted at 2024-05-01 20:10:15 on dist-test-slave-08p9"
I20240501 20:10:15.949400 16184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f856e0d1074b431eb19763847958b89e"
format_stamp: "Formatted at 2024-05-01 20:10:15 on dist-test-slave-08p9"
I20240501 20:10:15.953923 16184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.004s
I20240501 20:10:15.956912 16198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:15.958061 16184 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:15.958200 16184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f856e0d1074b431eb19763847958b89e"
format_stamp: "Formatted at 2024-05-01 20:10:15 on dist-test-slave-08p9"
I20240501 20:10:15.958406 16184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:15.973532 16184 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:15.974186 16184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:15.974463 16184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:15.975255 16184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:15.976495 16184 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:15.976580 16184 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:15.976680 16184 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:15.976747 16184 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:16.006002 16184 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:36919
I20240501 20:10:16.006059 16310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:36919 every 8 connection(s)
I20240501 20:10:16.007185 16184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:10:16.013332  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 16184
I20240501 20:10:16.013511  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:10:16.017076  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:16.019969 16311 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:16.020255 16311 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:16.021050 16311 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:16.022622 15734 ts_manager.cc:194] Registered new tserver with Master: f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:16.023510 15734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:42269
I20240501 20:10:16.148874 16051 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
W20240501 20:10:16.330173 16314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:16.330466 16314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:16.330600 16314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:16.338181 16314 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:16.338336 16314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:16.338524 16314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:10:16.347421 16314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:16.348577 16314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:16.350525 16314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:16.361995 16320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:16.362538 16319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:16.362879 16322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:16.363467 16314 server_base.cc:1034] running on GCE node
I20240501 20:10:16.364030 16314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:16.364866 16314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:16.366170 16314 hybrid_clock.cc:648] HybridClock initialized: now 1714594216366095 us; error 28 us; skew 500 ppm
I20240501 20:10:16.366668 16314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:16.368638 16314 webserver.cc:460] Webserver started at http://127.8.68.197:43167/ using document root <none> and password file <none>
I20240501 20:10:16.369297 16314 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:16.369402 16314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:16.369706 16314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:16.372093 16314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9a0f0037700246bdb65e05aef95fb2be"
format_stamp: "Formatted at 2024-05-01 20:10:16 on dist-test-slave-08p9"
I20240501 20:10:16.372795 16314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9a0f0037700246bdb65e05aef95fb2be"
format_stamp: "Formatted at 2024-05-01 20:10:16 on dist-test-slave-08p9"
I20240501 20:10:16.377473 16314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.005s
I20240501 20:10:16.380527 16328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:16.381697 16314 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:16.381837 16314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9a0f0037700246bdb65e05aef95fb2be"
format_stamp: "Formatted at 2024-05-01 20:10:16 on dist-test-slave-08p9"
I20240501 20:10:16.382005 16314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:16.417323 16314 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:16.417999 16314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:16.418241 16314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:16.419102 16314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:16.420389 16314 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:16.420476 16314 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:16.420575 16314 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:16.420643 16314 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:16.449012 16314 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:37623
I20240501 20:10:16.449103 16440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:37623 every 8 connection(s)
I20240501 20:10:16.450188 16314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:10:16.455334  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 16314
I20240501 20:10:16.455513  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:10:16.458696  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:0
--local_ip_for_outbound_sockets=127.8.68.198
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:16.461740 16441 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:16.461968 16441 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:16.462770 16441 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:16.463968 15734 ts_manager.cc:194] Registered new tserver with Master: 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623)
I20240501 20:10:16.464697 15734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:53003
I20240501 20:10:16.586452 16181 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
W20240501 20:10:16.775101 16444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:16.775342 16444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:16.775431 16444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:16.782374 16444 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:16.782469 16444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:16.782579 16444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240501 20:10:16.790954 16444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:16.792016 16444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:16.793519 16444 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:16.804505 16450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:16.805022 16449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:16.805402 16452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:16.805935 16444 server_base.cc:1034] running on GCE node
I20240501 20:10:16.806439 16444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:16.807195 16444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:16.808374 16444 hybrid_clock.cc:648] HybridClock initialized: now 1714594216808336 us; error 30 us; skew 500 ppm
I20240501 20:10:16.808691 16444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:16.810637 16444 webserver.cc:460] Webserver started at http://127.8.68.198:39375/ using document root <none> and password file <none>
I20240501 20:10:16.811242 16444 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:16.811345 16444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:16.811635 16444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:16.813835 16444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "d3917663631a4479b15355381486591c"
format_stamp: "Formatted at 2024-05-01 20:10:16 on dist-test-slave-08p9"
I20240501 20:10:16.814559 16444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "d3917663631a4479b15355381486591c"
format_stamp: "Formatted at 2024-05-01 20:10:16 on dist-test-slave-08p9"
I20240501 20:10:16.818962 16444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.001s
I20240501 20:10:16.821935 16458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:16.823182 16444 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 20:10:16.823315 16444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d3917663631a4479b15355381486591c"
format_stamp: "Formatted at 2024-05-01 20:10:16 on dist-test-slave-08p9"
I20240501 20:10:16.823477 16444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:16.861351 16444 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:16.862002 16444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:16.862221 16444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:16.863091 16444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:16.864410 16444 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:16.864490 16444 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:16.864588 16444 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:16.864650 16444 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:16.893283 16444 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:44869
I20240501 20:10:16.893368 16570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:44869 every 8 connection(s)
I20240501 20:10:16.894418 16444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:10:16.896790  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 16444
I20240501 20:10:16.896957  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:10:16.899886  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:0
--local_ip_for_outbound_sockets=127.8.68.199
--webserver_interface=127.8.68.199
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:16.905977 16571 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:16.906318 16571 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:16.907332 16571 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:16.913218 15734 ts_manager.cc:194] Registered new tserver with Master: d3917663631a4479b15355381486591c (127.8.68.198:44869)
I20240501 20:10:16.914156 15734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:49409
I20240501 20:10:17.025764 16311 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
W20240501 20:10:17.214390 16574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:17.214655 16574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:17.214805 16574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:17.222076 16574 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:17.222185 16574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:17.222368 16574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240501 20:10:17.231139 16574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:17.232272 16574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:17.233911 16574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:17.245436 16580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:17.245904 16579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:17.246311 16582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:17.246912 16574 server_base.cc:1034] running on GCE node
I20240501 20:10:17.247454 16574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:17.248286 16574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:17.249518 16574 hybrid_clock.cc:648] HybridClock initialized: now 1714594217249470 us; error 37 us; skew 500 ppm
I20240501 20:10:17.249823 16574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:17.252010 16574 webserver.cc:460] Webserver started at http://127.8.68.199:34917/ using document root <none> and password file <none>
I20240501 20:10:17.252668 16574 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:17.252772 16574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:17.253068 16574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:17.255429 16574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "cad00548f33849e7a19cecd760e1e429"
format_stamp: "Formatted at 2024-05-01 20:10:17 on dist-test-slave-08p9"
I20240501 20:10:17.256124 16574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "cad00548f33849e7a19cecd760e1e429"
format_stamp: "Formatted at 2024-05-01 20:10:17 on dist-test-slave-08p9"
I20240501 20:10:17.260692 16574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 20:10:17.263782 16588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:17.264945 16574 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:17.265100 16574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "cad00548f33849e7a19cecd760e1e429"
format_stamp: "Formatted at 2024-05-01 20:10:17 on dist-test-slave-08p9"
I20240501 20:10:17.265269 16574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:17.282915 16574 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:17.283605 16574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:17.283838 16574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:17.284653 16574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:17.285988 16574 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:17.286085 16574 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:17.286206 16574 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:17.286264 16574 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:17.314219 16574 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:44275
I20240501 20:10:17.314260 16700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:44275 every 8 connection(s)
I20240501 20:10:17.315371 16574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:10:17.320168  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 16574
I20240501 20:10:17.320344  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:10:17.323716  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.200:0
--local_ip_for_outbound_sockets=127.8.68.200
--webserver_interface=127.8.68.200
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:17.327682 16701 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:17.327957 16701 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:17.328820 16701 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:17.330251 15734 ts_manager.cc:194] Registered new tserver with Master: cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275)
I20240501 20:10:17.331059 15734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:58579
I20240501 20:10:17.466715 16441 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
W20240501 20:10:17.639616 16704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:17.639861 16704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:17.639989 16704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:17.646970 16704 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:17.647086 16704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:17.647223 16704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.200
I20240501 20:10:17.655545 16704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.68.200
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.200
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:17.656598 16704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:17.658107 16704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:17.669158 16709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:17.669644 16710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:17.670046 16712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:17.670574 16704 server_base.cc:1034] running on GCE node
I20240501 20:10:17.671089 16704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:17.671882 16704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:17.673102 16704 hybrid_clock.cc:648] HybridClock initialized: now 1714594217673059 us; error 34 us; skew 500 ppm
I20240501 20:10:17.673377 16704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:17.675338 16704 webserver.cc:460] Webserver started at http://127.8.68.200:37579/ using document root <none> and password file <none>
I20240501 20:10:17.675956 16704 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:17.676062 16704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:17.676344 16704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:17.678639 16704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "047275bda1074ac5a63325e7e1109c61"
format_stamp: "Formatted at 2024-05-01 20:10:17 on dist-test-slave-08p9"
I20240501 20:10:17.679332 16704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "047275bda1074ac5a63325e7e1109c61"
format_stamp: "Formatted at 2024-05-01 20:10:17 on dist-test-slave-08p9"
I20240501 20:10:17.683900 16704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:10:17.687165 16718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:17.688386 16704 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:10:17.688532 16704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "047275bda1074ac5a63325e7e1109c61"
format_stamp: "Formatted at 2024-05-01 20:10:17 on dist-test-slave-08p9"
I20240501 20:10:17.688699 16704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:17.705950 16704 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:17.706647 16704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:17.706878 16704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:17.707748 16704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:17.709095 16704 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:17.709179 16704 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:17.709276 16704 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:17.709340 16704 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:17.745216 16704 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.200:38791
I20240501 20:10:17.745301 16830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.200:38791 every 8 connection(s)
I20240501 20:10:17.746373 16704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:10:17.752975  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 16704
I20240501 20:10:17.753161  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:10:17.757973 16831 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:17.758212 16831 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:17.758929 16831 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:17.760108 15734 ts_manager.cc:194] Registered new tserver with Master: 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791)
I20240501 20:10:17.760807 15734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.200:43897
I20240501 20:10:17.768617  8467 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:10:17.783965  8467 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:10:17.784086  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 16314
I20240501 20:10:17.791332 16436 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:17.868126  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 16314
I20240501 20:10:17.881973  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 16444
I20240501 20:10:17.889436 16566 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:17.965678 16571 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
I20240501 20:10:17.967156  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 16444
W20240501 20:10:17.981645 15718 connection.cc:535] server connection from 127.8.68.198:49409 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:17.981887  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 16574
I20240501 20:10:17.989233 16696 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:18.068267  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 16574
I20240501 20:10:18.082311  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 16704
I20240501 20:10:18.089848 16826 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:18.167378  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 16704
I20240501 20:10:21.932636 15734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:21.992182 16245 tablet_service.cc:1467] Processing CreateTablet for tablet c74a6da428b04df58b7edcdf88d81cca (DEFAULT_TABLE table=rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.993736 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74a6da428b04df58b7edcdf88d81cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.993839 15986 tablet_service.cc:1467] Processing CreateTablet for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.993631 16246 tablet_service.cc:1467] Processing CreateTablet for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.995177 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.994836 15855 tablet_service.cc:1467] Processing CreateTablet for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.996111 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.996045 15985 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3eb13093b546d38fc71368f2083878 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.996938 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.997524 15985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3eb13093b546d38fc71368f2083878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.998371 15853 tablet_service.cc:1467] Processing CreateTablet for tablet c74a6da428b04df58b7edcdf88d81cca (DEFAULT_TABLE table=rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.998834 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74a6da428b04df58b7edcdf88d81cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.998423 15854 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3eb13093b546d38fc71368f2083878 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.999576 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3eb13093b546d38fc71368f2083878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.004395 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3eb13093b546d38fc71368f2083878 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.005651 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3eb13093b546d38fc71368f2083878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.010392 16115 tablet_service.cc:1467] Processing CreateTablet for tablet c74a6da428b04df58b7edcdf88d81cca (DEFAULT_TABLE table=rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.010928 16115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74a6da428b04df58b7edcdf88d81cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.014088 16860 tablet_bootstrap.cc:492] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:22.025611 16862 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:22.027349 16863 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:22.029948 16862 tablet_bootstrap.cc:654] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.031159 16863 tablet_bootstrap.cc:654] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.031239 16862 log.cc:824] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:22.032101 16860 tablet_bootstrap.cc:654] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.032626 16863 log.cc:824] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:22.032789 16861 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:22.033293 16860 log.cc:824] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:22.038108 16862 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:22.038537 16862 ts_tablet_manager.cc:1367] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.013s	user 0.006s	sys 0.004s
I20240501 20:10:22.039449 16861 tablet_bootstrap.cc:654] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.040639 16861 log.cc:824] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:22.041388 16863 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:22.041667 16863 ts_tablet_manager.cc:1367] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.004s
I20240501 20:10:22.048132 16860 tablet_bootstrap.cc:492] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:22.047880 16862 raft_consensus.cc:348] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.048362 16860 ts_tablet_manager.cc:1367] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.010s
I20240501 20:10:22.048519 16862 raft_consensus.cc:374] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.048686 16862 raft_consensus.cc:729] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.049494 16862 consensus_queue.cc:260] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.049808 16861 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:22.050045 16861 ts_tablet_manager.cc:1367] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.008s
I20240501 20:10:22.051282 16863 raft_consensus.cc:348] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.051741 16863 raft_consensus.cc:374] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.051851 16863 raft_consensus.cc:729] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.052640 16863 consensus_queue.cc:260] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.058071 16862 ts_tablet_manager.cc:1398] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.019s	user 0.010s	sys 0.008s
I20240501 20:10:22.058111 16863 ts_tablet_manager.cc:1398] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.016s	user 0.016s	sys 0.000s
I20240501 20:10:22.058529 16862 tablet_bootstrap.cc:492] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:22.058527 16863 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:22.059046 16860 raft_consensus.cc:348] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.059420 16860 raft_consensus.cc:374] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.059523 16860 raft_consensus.cc:729] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.060431 16862 tablet_bootstrap.cc:654] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.061419 16860 consensus_queue.cc:260] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.062078 16863 tablet_bootstrap.cc:654] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.065018 16861 raft_consensus.cc:348] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.065424 16861 raft_consensus.cc:374] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.065537 16861 raft_consensus.cc:729] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.066145 16861 consensus_queue.cc:260] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.068732 16863 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:22.068883 16863 ts_tablet_manager.cc:1367] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.004s
I20240501 20:10:22.069947 16863 raft_consensus.cc:348] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:22.070190 16863 raft_consensus.cc:374] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.070307 16863 raft_consensus.cc:729] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.070695 16863 consensus_queue.cc:260] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:22.070994 16860 ts_tablet_manager.cc:1398] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.023s	user 0.010s	sys 0.006s
I20240501 20:10:22.071378 16863 ts_tablet_manager.cc:1398] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:10:22.071650 16863 tablet_bootstrap.cc:492] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:22.073683 16860 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:22.073770 16863 tablet_bootstrap.cc:654] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.075371 16860 tablet_bootstrap.cc:654] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.076923 16862 tablet_bootstrap.cc:492] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:22.077123 16862 ts_tablet_manager.cc:1367] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.008s
I20240501 20:10:22.078091 16862 raft_consensus.cc:348] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.078375 16862 raft_consensus.cc:374] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.078505 16862 raft_consensus.cc:729] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.078836 16862 consensus_queue.cc:260] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.079398 16862 ts_tablet_manager.cc:1398] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:10:22.080178 16860 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:22.080206 16863 tablet_bootstrap.cc:492] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:22.080320 16860 ts_tablet_manager.cc:1367] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240501 20:10:22.080474 16863 ts_tablet_manager.cc:1367] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.007s
I20240501 20:10:22.081135 16863 raft_consensus.cc:348] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.081171 16860 raft_consensus.cc:348] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:22.081332 16860 raft_consensus.cc:374] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.081421 16860 raft_consensus.cc:729] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.081478 16861 ts_tablet_manager.cc:1398] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.031s	user 0.022s	sys 0.000s
I20240501 20:10:22.081454 16863 raft_consensus.cc:374] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.081629 16863 raft_consensus.cc:729] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.081678 16860 consensus_queue.cc:260] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:22.081871 16861 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:22.081987 16863 consensus_queue.cc:260] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.082137 16860 ts_tablet_manager.cc:1398] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:22.082628 16863 ts_tablet_manager.cc:1398] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:22.083370 16861 tablet_bootstrap.cc:654] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.086699 16861 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:22.086869 16861 ts_tablet_manager.cc:1367] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240501 20:10:22.087529 16861 raft_consensus.cc:348] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:22.087679 16861 raft_consensus.cc:374] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.087759 16861 raft_consensus.cc:729] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.087934 16861 consensus_queue.cc:260] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:22.088297 16861 ts_tablet_manager.cc:1398] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:10:22.108337 16868 raft_consensus.cc:482] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:22.108541 16868 raft_consensus.cc:504] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.110033 16868 leader_election.cc:288] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:22.116405 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25f3a86f0b494b89fe8f86aae4f4cd" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:22.116405 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25f3a86f0b494b89fe8f86aae4f4cd" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:22.116889 15875 raft_consensus.cc:2454] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:22.116884 16266 raft_consensus.cc:2454] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:22.117424 15939 leader_election.cc:302] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754fcd3b82f041568f546c6205f44272, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:22.117764 16868 raft_consensus.cc:2781] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:22.117880 16868 raft_consensus.cc:482] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:22.117967 16868 raft_consensus.cc:3037] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.119900 16868 raft_consensus.cc:504] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.120431 16868 leader_election.cc:288] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Requested vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:22.120760 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25f3a86f0b494b89fe8f86aae4f4cd" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:22.120838 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25f3a86f0b494b89fe8f86aae4f4cd" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:22.120906 16266 raft_consensus.cc:3037] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.120999 15875 raft_consensus.cc:3037] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.123123 16266 raft_consensus.cc:2454] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:22.123113 15875 raft_consensus.cc:2454] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:22.123500 15940 leader_election.cc:302] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:22.123798 16868 raft_consensus.cc:2781] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:22.124153 16868 raft_consensus.cc:686] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Becoming Leader. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Running, Role: LEADER
I20240501 20:10:22.124558 16868 consensus_queue.cc:237] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.128669 15734 catalog_manager.cc:5484] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754fcd3b82f041568f546c6205f44272 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:22.138638 16052 tablet.cc:2369] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:22.147701 16869 raft_consensus.cc:482] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:22.147835 16869 raft_consensus.cc:504] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:22.149186 16869 leader_election.cc:288] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:22.155673 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74a6da428b04df58b7edcdf88d81cca" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:22.155915 16266 raft_consensus.cc:2454] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:22.156332 15808 leader_election.cc:302] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:22.156272 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74a6da428b04df58b7edcdf88d81cca" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:22.156652 16869 raft_consensus.cc:2781] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:22.156699 16136 raft_consensus.cc:2454] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:22.156776 16869 raft_consensus.cc:482] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:22.156863 16869 raft_consensus.cc:3037] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.159258 16869 raft_consensus.cc:504] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:22.159904 16869 leader_election.cc:288] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Requested vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:22.160254 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74a6da428b04df58b7edcdf88d81cca" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:22.160415 16136 raft_consensus.cc:3037] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.160991 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74a6da428b04df58b7edcdf88d81cca" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:22.161176 16266 raft_consensus.cc:3037] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.163039 16266 raft_consensus.cc:2454] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:22.163354 15808 leader_election.cc:302] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:22.163357 16136 raft_consensus.cc:2454] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:22.163631 16869 raft_consensus.cc:2781] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:22.164052 16869 raft_consensus.cc:686] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Running, Role: LEADER
I20240501 20:10:22.164489 16869 consensus_queue.cc:237] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:22.168226 15734 catalog_manager.cc:5484] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:22.221865 15921 tablet.cc:2369] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:10:22.264658 16312 tablet.cc:2369] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:22.312642 16869 raft_consensus.cc:482] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:22.312844 16869 raft_consensus.cc:504] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.313493 16869 leader_election.cc:288] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:22.314013 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3eb13093b546d38fc71368f2083878" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:22.314303 16136 raft_consensus.cc:2454] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:22.314774 15810 leader_election.cc:302] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 
I20240501 20:10:22.315084 16869 raft_consensus.cc:2781] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:22.315218 16869 raft_consensus.cc:482] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:22.315351 16869 raft_consensus.cc:3037] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.318094 16869 raft_consensus.cc:504] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:22.318706 16869 leader_election.cc:288] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Requested vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:22.319119 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3eb13093b546d38fc71368f2083878" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:22.319356 16136 raft_consensus.cc:3037] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.320079 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3eb13093b546d38fc71368f2083878" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:22.320128 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3eb13093b546d38fc71368f2083878" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:22.320469 16006 raft_consensus.cc:2454] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:22.321444 16136 raft_consensus.cc:2454] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:22.321764 15810 leader_election.cc:302] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 754fcd3b82f041568f546c6205f44272
I20240501 20:10:22.322008 16869 raft_consensus.cc:2781] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:22.322196 16869 raft_consensus.cc:686] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Running, Role: LEADER
I20240501 20:10:22.322465 16869 consensus_queue.cc:237] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
W20240501 20:10:22.322871 16182 tablet.cc:2369] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:22.325093 15734 catalog_manager.cc:5484] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:22.547668 16868 consensus_queue.cc:1035] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.556210 16879 consensus_queue.cc:1035] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.608405 16869 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.613505 16869 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:22.654819 16886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:22.655097 16886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:22.662274 16886 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 20:10:22.791424 16005 raft_consensus.cc:3037] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.795681 16005 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.796394 16136 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.798761 16869 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.799263 16869 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.886474 16933 mvcc.cc:205] Tried to move back new op lower bound from 7022977936521760768 to 7022977934635261952. Current Snapshot: MvccSnapshot[applied={T|T < 7022977936521760768}]
I20240501 20:10:22.904356 16895 mvcc.cc:205] Tried to move back new op lower bound from 7022977936616456192 to 7022977934635261952. Current Snapshot: MvccSnapshot[applied={T|T < 7022977936616456192 or (T in {7022977936616456192})}]

INSERT report
    rows total: 2000
    time total: 216.026 ms
  time per row: 0.108013 ms
I20240501 20:10:23.009922 15731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:23.019393 15985 tablet_service.cc:1467] Processing CreateTablet for tablet 5682259bc54c469e9c6a08723c1bbee0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.019899 15985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5682259bc54c469e9c6a08723c1bbee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.020534 16245 tablet_service.cc:1467] Processing CreateTablet for tablet 58ae613944b941e694f790e02cfe5292 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.020952 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ae613944b941e694f790e02cfe5292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.024785 16954 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:23.026041 16954 tablet_bootstrap.cc:654] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.026695 16246 tablet_service.cc:1467] Processing CreateTablet for tablet 5682259bc54c469e9c6a08723c1bbee0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.027051 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5682259bc54c469e9c6a08723c1bbee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.027695 16115 tablet_service.cc:1467] Processing CreateTablet for tablet 58ae613944b941e694f790e02cfe5292 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.027895 16114 tablet_service.cc:1467] Processing CreateTablet for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.028121 16115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ae613944b941e694f790e02cfe5292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.028393 16114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.030989 15855 tablet_service.cc:1467] Processing CreateTablet for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.031463 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.032419 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 5682259bc54c469e9c6a08723c1bbee0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.032785 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5682259bc54c469e9c6a08723c1bbee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.036088 15986 tablet_service.cc:1467] Processing CreateTablet for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.036442 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.044337 16957 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:23.044818 15985 tablet_service.cc:1467] Processing CreateTablet for tablet 58ae613944b941e694f790e02cfe5292 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.046324 15985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ae613944b941e694f790e02cfe5292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.046672 16954 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:23.046852 16954 ts_tablet_manager.cc:1367] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.013s
I20240501 20:10:23.047475 16957 tablet_bootstrap.cc:654] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.047757 16954 raft_consensus.cc:348] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.048123 16954 raft_consensus.cc:374] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.048283 16954 raft_consensus.cc:729] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.048614 16954 consensus_queue.cc:260] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.049580 16954 ts_tablet_manager.cc:1398] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:10:23.050230 16954 tablet_bootstrap.cc:492] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:23.052699 16954 tablet_bootstrap.cc:654] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.053759 16956 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:23.055480 16956 tablet_bootstrap.cc:654] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.057078 16957 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:23.057194 16954 tablet_bootstrap.cc:492] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:23.057216 16957 ts_tablet_manager.cc:1367] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.013s	user 0.006s	sys 0.000s
I20240501 20:10:23.057339 16954 ts_tablet_manager.cc:1367] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.003s
I20240501 20:10:23.057952 16957 raft_consensus.cc:348] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:23.058145 16957 raft_consensus.cc:374] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.058044 16954 raft_consensus.cc:348] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.058221 16957 raft_consensus.cc:729] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.058241 16954 raft_consensus.cc:374] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.058326 16954 raft_consensus.cc:729] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.058539 16954 consensus_queue.cc:260] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.058656 16957 consensus_queue.cc:260] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:23.059028 16954 ts_tablet_manager.cc:1398] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:23.059322 16954 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:23.059906 16957 ts_tablet_manager.cc:1398] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:10:23.060232 16957 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:23.061838 16954 tablet_bootstrap.cc:654] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.062463 16957 tablet_bootstrap.cc:654] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.065464 16956 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:23.065522 16958 tablet_bootstrap.cc:492] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:23.065608 16956 ts_tablet_manager.cc:1367] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.004s
I20240501 20:10:23.066174 16957 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:23.066408 16957 ts_tablet_manager.cc:1367] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 20:10:23.066351 16956 raft_consensus.cc:348] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.066540 16956 raft_consensus.cc:374] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.066641 16956 raft_consensus.cc:729] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.066856 16956 consensus_queue.cc:260] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.067397 16956 ts_tablet_manager.cc:1398] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:23.067337 16957 raft_consensus.cc:348] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.067647 16957 raft_consensus.cc:374] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.067663 16956 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:23.067708 16958 tablet_bootstrap.cc:654] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.067800 16957 raft_consensus.cc:729] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.068105 16957 consensus_queue.cc:260] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.068637 16957 ts_tablet_manager.cc:1398] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:23.068951 16957 tablet_bootstrap.cc:492] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:23.070403 16954 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:23.070554 16954 ts_tablet_manager.cc:1367] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.000s
I20240501 20:10:23.070834 16957 tablet_bootstrap.cc:654] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.071326 16956 tablet_bootstrap.cc:654] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.072259 16954 raft_consensus.cc:348] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:23.072517 16954 raft_consensus.cc:374] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.072610 16954 raft_consensus.cc:729] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.072870 16954 consensus_queue.cc:260] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:23.073804 16954 ts_tablet_manager.cc:1398] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:10:23.075027 16958 tablet_bootstrap.cc:492] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:23.075173 16958 ts_tablet_manager.cc:1367] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.005s
I20240501 20:10:23.075935 16958 raft_consensus.cc:348] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.076140 16958 raft_consensus.cc:374] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.076267 16958 raft_consensus.cc:729] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.076507 16958 consensus_queue.cc:260] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.077087 16958 ts_tablet_manager.cc:1398] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:10:23.077996 16956 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:23.078152 16956 ts_tablet_manager.cc:1367] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240501 20:10:23.078238 16957 tablet_bootstrap.cc:492] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:23.078382 16957 ts_tablet_manager.cc:1367] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.001s
I20240501 20:10:23.078884 16957 raft_consensus.cc:348] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.078861 16956 raft_consensus.cc:348] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:23.079016 16957 raft_consensus.cc:374] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.079056 16956 raft_consensus.cc:374] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.079105 16957 raft_consensus.cc:729] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.079138 16956 raft_consensus.cc:729] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.079290 16957 consensus_queue.cc:260] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.079382 16956 consensus_queue.cc:260] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:23.079763 16957 ts_tablet_manager.cc:1398] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:10:23.079881 16956 ts_tablet_manager.cc:1398] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:23.128497 16893 raft_consensus.cc:482] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:23.128654 16893 raft_consensus.cc:504] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.130059 16893 leader_election.cc:288] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:23.137583 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5682259bc54c469e9c6a08723c1bbee0" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:23.137815 16006 raft_consensus.cc:2454] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:23.138263 16071 leader_election.cc:302] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:23.138259 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5682259bc54c469e9c6a08723c1bbee0" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:23.138464 16266 raft_consensus.cc:2454] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:23.138620 16893 raft_consensus.cc:2781] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:23.138729 16893 raft_consensus.cc:482] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:23.138828 16893 raft_consensus.cc:3037] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.140631 16893 raft_consensus.cc:504] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.141204 16893 leader_election.cc:288] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:23.141541 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5682259bc54c469e9c6a08723c1bbee0" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:23.141731 16006 raft_consensus.cc:3037] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.141788 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5682259bc54c469e9c6a08723c1bbee0" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:23.141963 16266 raft_consensus.cc:3037] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.143798 16006 raft_consensus.cc:2454] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:23.144165 16071 leader_election.cc:302] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:23.144394 16893 raft_consensus.cc:2781] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:23.144564 16266 raft_consensus.cc:2454] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:23.144939 16893 raft_consensus.cc:686] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Becoming Leader. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Running, Role: LEADER
I20240501 20:10:23.145356 16893 consensus_queue.cc:237] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.148785 15731 catalog_manager.cc:5484] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:23.161998 16928 raft_consensus.cc:482] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:23.162137 16928 raft_consensus.cc:504] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.162885 16928 leader_election.cc:288] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:23.163316 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fce37f2e8f74a119b8bd0c34b1a35ac" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:23.163359 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fce37f2e8f74a119b8bd0c34b1a35ac" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:23.163513 16136 raft_consensus.cc:2454] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:23.163543 16006 raft_consensus.cc:2454] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:23.163934 15810 leader_election.cc:302] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 
I20240501 20:10:23.164211 16928 raft_consensus.cc:2781] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:23.164323 16928 raft_consensus.cc:482] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:23.164407 16928 raft_consensus.cc:3037] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.166162 16928 raft_consensus.cc:504] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.166649 16928 leader_election.cc:288] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Requested vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:23.167057 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fce37f2e8f74a119b8bd0c34b1a35ac" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:23.167104 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fce37f2e8f74a119b8bd0c34b1a35ac" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:23.167243 16136 raft_consensus.cc:3037] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.167259 16006 raft_consensus.cc:3037] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.169787 16006 raft_consensus.cc:2454] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:23.169802 16136 raft_consensus.cc:2454] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:23.170147 15810 leader_election.cc:302] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 
I20240501 20:10:23.170421 16928 raft_consensus.cc:2781] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:23.170580 16928 raft_consensus.cc:686] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Running, Role: LEADER
I20240501 20:10:23.170835 16928 consensus_queue.cc:237] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.173210 15731 catalog_manager.cc:5484] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:23.208032 16893 raft_consensus.cc:482] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:23.208182 16893 raft_consensus.cc:504] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:23.208691 16893 leader_election.cc:288] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:23.209115 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ae613944b941e694f790e02cfe5292" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:23.209275 16006 raft_consensus.cc:2454] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:23.209293 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ae613944b941e694f790e02cfe5292" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:23.209528 16266 raft_consensus.cc:2454] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:23.209579 16071 leader_election.cc:302] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:23.209805 16893 raft_consensus.cc:2781] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:23.209880 16893 raft_consensus.cc:482] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:23.209980 16893 raft_consensus.cc:3037] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.211735 16893 raft_consensus.cc:504] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:23.212234 16893 leader_election.cc:288] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:23.212528 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ae613944b941e694f790e02cfe5292" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:23.212713 16006 raft_consensus.cc:3037] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.212679 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ae613944b941e694f790e02cfe5292" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:23.212872 16266 raft_consensus.cc:3037] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.215265 16006 raft_consensus.cc:2454] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:23.215476 16266 raft_consensus.cc:2454] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:23.215632 16071 leader_election.cc:302] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:23.215881 16893 raft_consensus.cc:2781] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:23.216023 16893 raft_consensus.cc:686] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Becoming Leader. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Running, Role: LEADER
I20240501 20:10:23.216245 16893 consensus_queue.cc:237] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:23.218492 15731 catalog_manager.cc:5484] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:23.548391 16974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:23.548645 16974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:23.555680 16974 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:10:23.657873 16893 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.666931 17001 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.673820 16928 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.705657 16928 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.718614 16006 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.719234 16893 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.720224 16266 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.722095 16893 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.754709 17000 mvcc.cc:205] Tried to move back new op lower bound from 7022977940345962496 to 7022977938295599104. Current Snapshot: MvccSnapshot[applied={T|T < 7022977940345962496}]

INSERT report
    rows total: 2000
    time total: 235.029 ms
  time per row: 0.117514 ms
I20240501 20:10:23.926968 15731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:23.935364 15855 tablet_service.cc:1467] Processing CreateTablet for tablet f3761f02c20a4448acd89f4a37f1193c (DEFAULT_TABLE table=rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.935840 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3761f02c20a4448acd89f4a37f1193c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.936373 15853 tablet_service.cc:1467] Processing CreateTablet for tablet e483217b8761457d84edc49253877719 (DEFAULT_TABLE table=rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.936599 16114 tablet_service.cc:1467] Processing CreateTablet for tablet e483217b8761457d84edc49253877719 (DEFAULT_TABLE table=rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.936825 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e483217b8761457d84edc49253877719. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.936990 16114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e483217b8761457d84edc49253877719. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.936998 15854 tablet_service.cc:1467] Processing CreateTablet for tablet 627e9e5c120747f1916855ea83008657 (DEFAULT_TABLE table=rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.937371 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627e9e5c120747f1916855ea83008657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.939206 15985 tablet_service.cc:1467] Processing CreateTablet for tablet f3761f02c20a4448acd89f4a37f1193c (DEFAULT_TABLE table=rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.939272 16244 tablet_service.cc:1467] Processing CreateTablet for tablet 627e9e5c120747f1916855ea83008657 (DEFAULT_TABLE table=rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.939599 15985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3761f02c20a4448acd89f4a37f1193c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.939677 16244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627e9e5c120747f1916855ea83008657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.942643 16246 tablet_service.cc:1467] Processing CreateTablet for tablet f3761f02c20a4448acd89f4a37f1193c (DEFAULT_TABLE table=rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.943019 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3761f02c20a4448acd89f4a37f1193c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.945503 16245 tablet_service.cc:1467] Processing CreateTablet for tablet e483217b8761457d84edc49253877719 (DEFAULT_TABLE table=rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.945827 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e483217b8761457d84edc49253877719. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.950119 17044 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:23.950872 17042 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:23.951856 17044 tablet_bootstrap.cc:654] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.952109 15986 tablet_service.cc:1467] Processing CreateTablet for tablet 627e9e5c120747f1916855ea83008657 (DEFAULT_TABLE table=rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.952423 17042 tablet_bootstrap.cc:654] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.952442 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627e9e5c120747f1916855ea83008657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.957819 17043 tablet_bootstrap.cc:492] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:23.961071 17043 tablet_bootstrap.cc:654] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.963027 17042 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:23.963166 17042 ts_tablet_manager.cc:1367] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.006s
I20240501 20:10:23.963294 17043 tablet_bootstrap.cc:492] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:23.963425 17043 ts_tablet_manager.cc:1367] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240501 20:10:23.963881 17042 raft_consensus.cc:348] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.964102 17042 raft_consensus.cc:374] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.964098 17043 raft_consensus.cc:348] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.964197 17042 raft_consensus.cc:729] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.964414 17042 consensus_queue.cc:260] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.964972 17042 ts_tablet_manager.cc:1398] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:10:23.965241 17042 tablet_bootstrap.cc:492] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:23.966679 17042 tablet_bootstrap.cc:654] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.969077 17047 tablet_bootstrap.cc:492] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:23.969913 17042 tablet_bootstrap.cc:492] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:23.970068 17042 ts_tablet_manager.cc:1367] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 20:10:23.970724 17047 tablet_bootstrap.cc:654] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.970791 17042 raft_consensus.cc:348] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.971001 17042 raft_consensus.cc:374] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.971081 17042 raft_consensus.cc:729] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.971329 17043 raft_consensus.cc:374] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.971314 17042 consensus_queue.cc:260] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.971442 17043 raft_consensus.cc:729] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.971810 17042 ts_tablet_manager.cc:1398] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:23.972097 17042 tablet_bootstrap.cc:492] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:23.973637 17042 tablet_bootstrap.cc:654] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.975147 17043 consensus_queue.cc:260] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.975576 17044 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:23.975694 17043 ts_tablet_manager.cc:1398] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240501 20:10:23.975796 17044 ts_tablet_manager.cc:1367] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.010s
I20240501 20:10:23.976035 17043 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:23.976598 17044 raft_consensus.cc:348] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.976846 17044 raft_consensus.cc:374] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.976967 17044 raft_consensus.cc:729] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.977267 17044 consensus_queue.cc:260] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.977413 17043 tablet_bootstrap.cc:654] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.977959 17044 ts_tablet_manager.cc:1398] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:10:23.979952 17044 tablet_bootstrap.cc:492] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:23.981570 17044 tablet_bootstrap.cc:654] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.984144 17042 tablet_bootstrap.cc:492] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:23.984287 17042 ts_tablet_manager.cc:1367] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240501 20:10:23.985005 17042 raft_consensus.cc:348] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.985234 17042 raft_consensus.cc:374] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.985316 17042 raft_consensus.cc:729] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.985524 17042 consensus_queue.cc:260] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.985827 17047 tablet_bootstrap.cc:492] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:23.985985 17047 ts_tablet_manager.cc:1367] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.008s
I20240501 20:10:23.986006 17042 ts_tablet_manager.cc:1398] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:23.986877 17044 tablet_bootstrap.cc:492] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:23.986865 17047 raft_consensus.cc:348] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.987092 17043 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:23.987087 17047 raft_consensus.cc:374] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.987135 17044 ts_tablet_manager.cc:1367] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.001s
I20240501 20:10:23.987186 17047 raft_consensus.cc:729] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.987249 17043 ts_tablet_manager.cc:1367] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.005s
I20240501 20:10:23.987398 17047 consensus_queue.cc:260] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.988060 17047 ts_tablet_manager.cc:1398] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:23.987936 17043 raft_consensus.cc:348] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.988170 17043 raft_consensus.cc:374] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.988005 17044 raft_consensus.cc:348] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.988248 17043 raft_consensus.cc:729] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.988334 17044 raft_consensus.cc:374] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.988440 17044 raft_consensus.cc:729] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.988488 17043 consensus_queue.cc:260] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.988673 17044 consensus_queue.cc:260] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:23.989109 17043 ts_tablet_manager.cc:1398] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:10:23.989145 17044 ts_tablet_manager.cc:1398] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:10:23.989459 17044 tablet_bootstrap.cc:492] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:23.991055 17044 tablet_bootstrap.cc:654] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.992606 17044 tablet_bootstrap.cc:492] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:23.992851 17044 ts_tablet_manager.cc:1367] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.003s
I20240501 20:10:23.993410 17044 raft_consensus.cc:348] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.993585 17044 raft_consensus.cc:374] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.993678 17044 raft_consensus.cc:729] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.993882 17044 consensus_queue.cc:260] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:23.994410 17044 ts_tablet_manager.cc:1398] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I20240501 20:10:24.003546 16946 raft_consensus.cc:482] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:24.003661 16946 raft_consensus.cc:504] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.004226 16946 leader_election.cc:288] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:24.004580 15874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3761f02c20a4448acd89f4a37f1193c" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:24.004630 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3761f02c20a4448acd89f4a37f1193c" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:24.004770 15874 raft_consensus.cc:2454] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:24.004776 16265 raft_consensus.cc:2454] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:24.005113 15939 leader_election.cc:302] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754fcd3b82f041568f546c6205f44272, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:24.005336 16946 raft_consensus.cc:2781] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:24.005431 16946 raft_consensus.cc:482] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:24.005527 16946 raft_consensus.cc:3037] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.007336 16946 raft_consensus.cc:504] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.008278 15874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3761f02c20a4448acd89f4a37f1193c" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:24.008396 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3761f02c20a4448acd89f4a37f1193c" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:24.008459 15874 raft_consensus.cc:3037] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.008597 16265 raft_consensus.cc:3037] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.011147 15874 raft_consensus.cc:2454] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:24.011371 16265 raft_consensus.cc:2454] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:24.011512 15940 leader_election.cc:302] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:24.011606 16946 leader_election.cc:288] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Requested vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:24.011914 16946 raft_consensus.cc:2781] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:24.012038 16946 raft_consensus.cc:686] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Becoming Leader. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Running, Role: LEADER
I20240501 20:10:24.012315 16946 consensus_queue.cc:237] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.015300 15731 catalog_manager.cc:5484] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754fcd3b82f041568f546c6205f44272 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:24.078923 17031 raft_consensus.cc:482] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:24.079097 17031 raft_consensus.cc:504] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.079720 17031 leader_election.cc:288] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:24.080092 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627e9e5c120747f1916855ea83008657" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:24.080168 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627e9e5c120747f1916855ea83008657" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:24.080289 16265 raft_consensus.cc:2454] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:24.080386 16005 raft_consensus.cc:2454] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:24.080663 15808 leader_election.cc:302] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:24.080940 17031 raft_consensus.cc:2781] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:24.081063 17031 raft_consensus.cc:482] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:24.081153 17031 raft_consensus.cc:3037] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.083021 17031 raft_consensus.cc:504] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.083493 17031 leader_election.cc:288] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Requested vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:24.083832 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627e9e5c120747f1916855ea83008657" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:24.083909 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "627e9e5c120747f1916855ea83008657" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:24.084003 16265 raft_consensus.cc:3037] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.084087 16005 raft_consensus.cc:3037] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.085850 16265 raft_consensus.cc:2454] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:24.085909 16005 raft_consensus.cc:2454] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:24.086233 15810 leader_election.cc:302] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:24.086556 17031 raft_consensus.cc:2781] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:24.086690 17031 raft_consensus.cc:686] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Running, Role: LEADER
I20240501 20:10:24.086951 17031 consensus_queue.cc:237] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.089525 15731 catalog_manager.cc:5484] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:24.125864 17031 raft_consensus.cc:482] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:24.126031 17031 raft_consensus.cc:504] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.126574 17031 leader_election.cc:288] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:24.127094 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e483217b8761457d84edc49253877719" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:24.127122 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e483217b8761457d84edc49253877719" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:24.127321 16265 raft_consensus.cc:2454] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:24.127342 16136 raft_consensus.cc:2454] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:24.127705 15810 leader_election.cc:302] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:24.127970 17031 raft_consensus.cc:2781] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:24.128054 17031 raft_consensus.cc:482] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:24.128152 17031 raft_consensus.cc:3037] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.129936 17031 raft_consensus.cc:504] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.130466 17031 leader_election.cc:288] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Requested vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:24.130825 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e483217b8761457d84edc49253877719" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:24.130844 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e483217b8761457d84edc49253877719" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:24.130962 16265 raft_consensus.cc:3037] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.131039 16136 raft_consensus.cc:3037] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.132979 16136 raft_consensus.cc:2454] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:24.133112 16265 raft_consensus.cc:2454] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:24.133333 15810 leader_election.cc:302] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 
I20240501 20:10:24.133603 17031 raft_consensus.cc:2781] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:24.133885 17031 raft_consensus.cc:686] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Running, Role: LEADER
I20240501 20:10:24.134138 17031 consensus_queue.cc:237] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.136474 15731 catalog_manager.cc:5484] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:24.496685 17059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:24.496953 17059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:24.503856 17059 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:24.524052 17031 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:24.530606 17031 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 20:10:24.551394 17031 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:24.555632 17031 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:24.591864 17055 consensus_queue.cc:1035] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:24.598109 17093 consensus_queue.cc:1035] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 257.479 ms
  time per row: 0.12874 ms
I20240501 20:10:24.900957 15731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:24.909372 15986 tablet_service.cc:1467] Processing CreateTablet for tablet f638f341ff2341dc82889334eb87e3d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.909610 16114 tablet_service.cc:1467] Processing CreateTablet for tablet f638f341ff2341dc82889334eb87e3d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.909832 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f638f341ff2341dc82889334eb87e3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.910019 16114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f638f341ff2341dc82889334eb87e3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.910878 16245 tablet_service.cc:1467] Processing CreateTablet for tablet 98eb95ded28d433aad9523586e29947f (DEFAULT_TABLE table=rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.911044 16244 tablet_service.cc:1467] Processing CreateTablet for tablet f904b62401ff4528b3fa65dd4e11512a (DEFAULT_TABLE table=rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.911268 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98eb95ded28d433aad9523586e29947f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.912104 15985 tablet_service.cc:1467] Processing CreateTablet for tablet f904b62401ff4528b3fa65dd4e11512a (DEFAULT_TABLE table=rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.912432 15985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f904b62401ff4528b3fa65dd4e11512a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.912775 15854 tablet_service.cc:1467] Processing CreateTablet for tablet f638f341ff2341dc82889334eb87e3d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.913172 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f638f341ff2341dc82889334eb87e3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.915207 15853 tablet_service.cc:1467] Processing CreateTablet for tablet 98eb95ded28d433aad9523586e29947f (DEFAULT_TABLE table=rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.915549 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98eb95ded28d433aad9523586e29947f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.917500 17127 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:24.918949 16115 tablet_service.cc:1467] Processing CreateTablet for tablet 98eb95ded28d433aad9523586e29947f (DEFAULT_TABLE table=rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.919045 16116 tablet_service.cc:1467] Processing CreateTablet for tablet f904b62401ff4528b3fa65dd4e11512a (DEFAULT_TABLE table=rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.919147 17127 tablet_bootstrap.cc:654] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.919324 16115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98eb95ded28d433aad9523586e29947f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.919611 16244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f904b62401ff4528b3fa65dd4e11512a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.922534 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f904b62401ff4528b3fa65dd4e11512a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.924230 17127 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:24.924383 17127 ts_tablet_manager.cc:1367] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.007s
I20240501 20:10:24.925187 17127 raft_consensus.cc:348] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:24.926483 17127 raft_consensus.cc:374] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.926599 17127 raft_consensus.cc:729] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.926851 17127 consensus_queue.cc:260] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:24.927532 17127 ts_tablet_manager.cc:1398] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:10:24.934410 17127 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:24.934420 17130 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:24.934695 17128 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:24.935827 17127 tablet_bootstrap.cc:654] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.936008 17130 tablet_bootstrap.cc:654] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.936750 17128 tablet_bootstrap.cc:654] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.937327 17126 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:24.938912 17127 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:24.939100 17126 tablet_bootstrap.cc:654] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.939100 17127 ts_tablet_manager.cc:1367] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 20:10:24.939316 17128 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:24.939457 17128 ts_tablet_manager.cc:1367] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 20:10:24.939700 17127 raft_consensus.cc:348] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.939869 17127 raft_consensus.cc:374] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.939975 17127 raft_consensus.cc:729] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.940196 17128 raft_consensus.cc:348] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.940224 17127 consensus_queue.cc:260] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.940433 17128 raft_consensus.cc:374] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.940558 17128 raft_consensus.cc:729] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.940680 17130 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:24.940721 17127 ts_tablet_manager.cc:1398] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:10:24.940832 17130 ts_tablet_manager.cc:1367] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240501 20:10:24.940821 17128 consensus_queue.cc:260] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.941534 17130 raft_consensus.cc:348] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.941732 17130 raft_consensus.cc:374] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.941830 17130 raft_consensus.cc:729] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.942037 17130 consensus_queue.cc:260] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.942433 17128 ts_tablet_manager.cc:1398] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:10:24.942551 17130 ts_tablet_manager.cc:1398] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:24.942718 17128 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:24.943703 17126 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:24.943861 17126 ts_tablet_manager.cc:1367] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.003s
I20240501 20:10:24.944146 17128 tablet_bootstrap.cc:654] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.944325 17130 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:24.944638 17126 raft_consensus.cc:348] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.944846 17126 raft_consensus.cc:374] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.944952 17126 raft_consensus.cc:729] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.945212 17126 consensus_queue.cc:260] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.945727 17130 tablet_bootstrap.cc:654] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.946430 17126 ts_tablet_manager.cc:1398] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:10:24.946761 17126 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:24.948276 17126 tablet_bootstrap.cc:654] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.953189 17126 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:24.953394 17126 ts_tablet_manager.cc:1367] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.002s
I20240501 20:10:24.954241 17126 raft_consensus.cc:348] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.954649 17126 raft_consensus.cc:374] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.954762 17126 raft_consensus.cc:729] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.955184 17126 consensus_queue.cc:260] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.956231 17126 ts_tablet_manager.cc:1398] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240501 20:10:24.957404 17128 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:24.957557 17130 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:24.957561 17128 ts_tablet_manager.cc:1367] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.015s	user 0.001s	sys 0.013s
I20240501 20:10:24.958216 17130 ts_tablet_manager.cc:1367] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.011s
I20240501 20:10:24.958305 17128 raft_consensus.cc:348] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:24.958504 17128 raft_consensus.cc:374] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.958580 17128 raft_consensus.cc:729] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.958813 17128 consensus_queue.cc:260] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:24.958925 17130 raft_consensus.cc:348] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:24.959131 17130 raft_consensus.cc:374] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.959223 17130 raft_consensus.cc:729] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.959349 17128 ts_tablet_manager.cc:1398] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:10:24.959477 17130 consensus_queue.cc:260] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:24.960425 17130 ts_tablet_manager.cc:1398] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:10:24.960657 17130 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:24.961907 17130 tablet_bootstrap.cc:654] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.964207 17130 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:24.964344 17130 ts_tablet_manager.cc:1367] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.002s
I20240501 20:10:24.964871 17130 raft_consensus.cc:348] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.965063 17130 raft_consensus.cc:374] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.965157 17130 raft_consensus.cc:729] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.965329 17130 consensus_queue.cc:260] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:24.965713 17130 ts_tablet_manager.cc:1398] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:10:24.967368 17055 raft_consensus.cc:482] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:24.967504 17055 raft_consensus.cc:504] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.968153 17055 leader_election.cc:288] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:24.968544 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f638f341ff2341dc82889334eb87e3d0" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:24.968747 15875 raft_consensus.cc:2454] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:24.969188 15940 leader_election.cc:302] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:24.969492 17055 raft_consensus.cc:2781] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:24.969588 17055 raft_consensus.cc:482] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:24.969694 17055 raft_consensus.cc:3037] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.971566 17055 raft_consensus.cc:504] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.972476 17055 leader_election.cc:288] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Requested vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:24.972491 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f638f341ff2341dc82889334eb87e3d0" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:24.972697 15875 raft_consensus.cc:3037] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.975327 15875 raft_consensus.cc:2454] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:24.975659 15940 leader_election.cc:302] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:24.975901 17055 raft_consensus.cc:2781] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:24.975836 16135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f638f341ff2341dc82889334eb87e3d0" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:24.976020 16135 raft_consensus.cc:2454] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:24.976065 17055 raft_consensus.cc:686] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Becoming Leader. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Running, Role: LEADER
I20240501 20:10:24.976346 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f638f341ff2341dc82889334eb87e3d0" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:24.976497 16136 raft_consensus.cc:3037] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.976442 17055 consensus_queue.cc:237] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:24.978310 16136 raft_consensus.cc:2454] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:24.978883 15731 catalog_manager.cc:5484] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754fcd3b82f041568f546c6205f44272 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:25.009528 17070 raft_consensus.cc:482] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.009681 17070 raft_consensus.cc:504] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.011286 17070 leader_election.cc:288] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:25.018159 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98eb95ded28d433aad9523586e29947f" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:25.018412 16136 raft_consensus.cc:2454] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:25.018852 16201 leader_election.cc:302] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:25.019244 17070 raft_consensus.cc:2781] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:25.019364 17070 raft_consensus.cc:482] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:25.019448 17070 raft_consensus.cc:3037] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.019527 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98eb95ded28d433aad9523586e29947f" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:25.019724 15875 raft_consensus.cc:2454] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:25.021723 17070 raft_consensus.cc:504] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.022233 17070 leader_election.cc:288] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Requested vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:25.022778 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98eb95ded28d433aad9523586e29947f" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:25.022763 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98eb95ded28d433aad9523586e29947f" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:25.022938 15875 raft_consensus.cc:3037] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.022954 16136 raft_consensus.cc:3037] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.025563 16136 raft_consensus.cc:2454] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:25.025579 15875 raft_consensus.cc:2454] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:25.025911 16201 leader_election.cc:302] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:25.026191 17070 raft_consensus.cc:2781] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:25.026608 17070 raft_consensus.cc:686] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Becoming Leader. State: Replica: f856e0d1074b431eb19763847958b89e, State: Running, Role: LEADER
I20240501 20:10:25.027174 17070 consensus_queue.cc:237] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.028155 17055 raft_consensus.cc:482] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.028312 17055 raft_consensus.cc:504] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:25.028968 17055 leader_election.cc:288] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:25.029518 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f904b62401ff4528b3fa65dd4e11512a" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:25.029628 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f904b62401ff4528b3fa65dd4e11512a" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:25.029781 16266 raft_consensus.cc:2454] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:25.029817 16136 raft_consensus.cc:2454] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:25.030171 15941 leader_election.cc:302] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:25.030459 17055 raft_consensus.cc:2781] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:25.030577 17055 raft_consensus.cc:482] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:25.030658 17055 raft_consensus.cc:3037] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.030848 15731 catalog_manager.cc:5484] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e reported cstate change: term changed from 0 to 1, leader changed from <none> to f856e0d1074b431eb19763847958b89e (127.8.68.196). New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:25.033227 17055 raft_consensus.cc:504] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:25.033794 17055 leader_election.cc:288] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Requested vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:25.034153 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f904b62401ff4528b3fa65dd4e11512a" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:25.034196 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f904b62401ff4528b3fa65dd4e11512a" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:25.034360 16136 raft_consensus.cc:3037] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.034365 16266 raft_consensus.cc:3037] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.036065 16136 raft_consensus.cc:2454] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:25.036387 15941 leader_election.cc:302] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:25.036902 16266 raft_consensus.cc:2454] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:25.037151 17055 raft_consensus.cc:2781] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:25.037295 17055 raft_consensus.cc:686] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Becoming Leader. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Running, Role: LEADER
I20240501 20:10:25.037528 17055 consensus_queue.cc:237] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:25.039722 15731 catalog_manager.cc:5484] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754fcd3b82f041568f546c6205f44272 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:25.367588 17147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:25.367866 17147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:25.374994 17147 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 20:10:25.409721 17093 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.415967 17055 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.430511 17166 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.434741 17055 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.464057 17070 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.466598 17070 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 231.588 ms
  time per row: 0.115794 ms
I20240501 20:10:25.758744 15729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:25.767107 16114 tablet_service.cc:1467] Processing CreateTablet for tablet f7fb9bc4cd73492da44bde1eecd627dc (DEFAULT_TABLE table=rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.767563 16114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7fb9bc4cd73492da44bde1eecd627dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.768141 15854 tablet_service.cc:1467] Processing CreateTablet for tablet f7fb9bc4cd73492da44bde1eecd627dc (DEFAULT_TABLE table=rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.768410 15853 tablet_service.cc:1467] Processing CreateTablet for tablet d48de8e0c12b432ea5079390400302c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.768586 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7fb9bc4cd73492da44bde1eecd627dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.768822 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48de8e0c12b432ea5079390400302c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.769744 15855 tablet_service.cc:1467] Processing CreateTablet for tablet 1474070bbdf147d2ae3313fdf9d71c19 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.770141 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1474070bbdf147d2ae3313fdf9d71c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.772791 15986 tablet_service.cc:1467] Processing CreateTablet for tablet f7fb9bc4cd73492da44bde1eecd627dc (DEFAULT_TABLE table=rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.773198 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7fb9bc4cd73492da44bde1eecd627dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.776748 17214 tablet_bootstrap.cc:492] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:25.778358 17214 tablet_bootstrap.cc:654] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.778643 15985 tablet_service.cc:1467] Processing CreateTablet for tablet d48de8e0c12b432ea5079390400302c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.778990 15985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48de8e0c12b432ea5079390400302c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.782985 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 1474070bbdf147d2ae3313fdf9d71c19 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.783383 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1474070bbdf147d2ae3313fdf9d71c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.783986 16244 tablet_service.cc:1467] Processing CreateTablet for tablet d48de8e0c12b432ea5079390400302c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.784093 16245 tablet_service.cc:1467] Processing CreateTablet for tablet 1474070bbdf147d2ae3313fdf9d71c19 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.784391 16244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48de8e0c12b432ea5079390400302c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.784608 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1474070bbdf147d2ae3313fdf9d71c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.792285 17215 tablet_bootstrap.cc:492] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:25.792971 17213 tablet_bootstrap.cc:492] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:25.794049 17215 tablet_bootstrap.cc:654] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.794857 17213 tablet_bootstrap.cc:654] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.797437 17216 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:25.799106 17216 tablet_bootstrap.cc:654] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.801559 17213 tablet_bootstrap.cc:492] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:25.801702 17213 ts_tablet_manager.cc:1367] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I20240501 20:10:25.802083 17215 tablet_bootstrap.cc:492] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:25.802219 17215 ts_tablet_manager.cc:1367] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240501 20:10:25.802508 17213 raft_consensus.cc:348] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.802719 17213 raft_consensus.cc:374] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.802819 17213 raft_consensus.cc:729] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.802953 17215 raft_consensus.cc:348] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.803169 17215 raft_consensus.cc:374] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.803079 17213 consensus_queue.cc:260] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.803282 17215 raft_consensus.cc:729] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.803807 17213 ts_tablet_manager.cc:1398] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:25.804118 17213 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:25.804595 17215 consensus_queue.cc:260] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.805747 17215 ts_tablet_manager.cc:1398] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 20:10:25.805996 17215 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:25.806308 17213 tablet_bootstrap.cc:654] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.807412 17215 tablet_bootstrap.cc:654] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.812067 17216 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:25.812333 17216 ts_tablet_manager.cc:1367] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.004s
I20240501 20:10:25.813021 17216 raft_consensus.cc:348] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:25.813202 17216 raft_consensus.cc:374] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.813289 17216 raft_consensus.cc:729] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.813504 17216 consensus_queue.cc:260] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:25.814003 17216 ts_tablet_manager.cc:1398] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:10:25.814252 17216 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:25.815819 17216 tablet_bootstrap.cc:654] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.817245 17188 raft_consensus.cc:482] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.817378 17188 raft_consensus.cc:504] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.818034 17188 leader_election.cc:288] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:25.818640 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fb9bc4cd73492da44bde1eecd627dc" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:25.819301 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fb9bc4cd73492da44bde1eecd627dc" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:25.819507 16134 raft_consensus.cc:2454] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:25.820035 15941 leader_election.cc:302] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:25.820384 17188 raft_consensus.cc:2781] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:25.820487 17188 raft_consensus.cc:482] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:25.820583 17188 raft_consensus.cc:3037] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.823341 17188 raft_consensus.cc:504] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
W20240501 20:10:25.824445 15940 leader_election.cc:341] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:25.825186 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fb9bc4cd73492da44bde1eecd627dc" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:25.825381 16134 raft_consensus.cc:3037] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.827706 17215 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:25.827845 17215 ts_tablet_manager.cc:1367] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.003s
I20240501 20:10:25.827878 16134 raft_consensus.cc:2454] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:25.828208 15941 leader_election.cc:302] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:25.828518 17166 raft_consensus.cc:2781] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:25.828544 17215 raft_consensus.cc:348] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:25.828667 17166 raft_consensus.cc:686] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Becoming Leader. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Running, Role: LEADER
I20240501 20:10:25.828790 17215 raft_consensus.cc:374] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.828882 17215 raft_consensus.cc:729] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.828941 17166 consensus_queue.cc:237] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.829033 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fb9bc4cd73492da44bde1eecd627dc" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:25.829115 17215 consensus_queue.cc:260] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
W20240501 20:10:25.829576 15940 leader_election.cc:341] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:25.829632 17215 ts_tablet_manager.cc:1398] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:25.829792 17188 leader_election.cc:288] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Requested vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:25.830864 17214 tablet_bootstrap.cc:492] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:25.830891 17213 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:25.831036 17213 ts_tablet_manager.cc:1367] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.027s	user 0.002s	sys 0.017s
I20240501 20:10:25.831063 17214 ts_tablet_manager.cc:1367] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.054s	user 0.014s	sys 0.005s
I20240501 20:10:25.831750 17213 raft_consensus.cc:348] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:25.831956 17213 raft_consensus.cc:374] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.832044 17213 raft_consensus.cc:729] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.832063 17214 raft_consensus.cc:348] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.832337 17214 raft_consensus.cc:374] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.832301 17213 consensus_queue.cc:260] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:25.832525 17214 raft_consensus.cc:729] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.832906 17213 ts_tablet_manager.cc:1398] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:10:25.832892 17214 consensus_queue.cc:260] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.833635 17214 ts_tablet_manager.cc:1398] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:25.833925 17214 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:25.834858 17216 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:25.835070 17216 ts_tablet_manager.cc:1367] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.021s	user 0.006s	sys 0.002s
I20240501 20:10:25.835307 17214 tablet_bootstrap.cc:654] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.836511 17216 raft_consensus.cc:348] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:25.836841 17216 raft_consensus.cc:374] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.836953 17216 raft_consensus.cc:729] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.837272 17216 consensus_queue.cc:260] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:25.837826 17216 ts_tablet_manager.cc:1398] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 20:10:25.840229 15730 catalog_manager.cc:5484] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754fcd3b82f041568f546c6205f44272 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:25.843230 17214 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:25.843381 17214 ts_tablet_manager.cc:1367] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.009s	user 0.008s	sys 0.000s
I20240501 20:10:25.844416 17214 raft_consensus.cc:348] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:25.844628 17214 raft_consensus.cc:374] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.844699 17214 raft_consensus.cc:729] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.844904 17214 consensus_queue.cc:260] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:25.845403 17214 ts_tablet_manager.cc:1398] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:25.845656 17214 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:25.847062 17214 tablet_bootstrap.cc:654] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.848778 17214 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:25.848913 17214 ts_tablet_manager.cc:1367] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:10:25.849640 17214 raft_consensus.cc:348] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:25.849831 17214 raft_consensus.cc:374] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.849915 17214 raft_consensus.cc:729] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.850107 17214 consensus_queue.cc:260] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:25.850661 17214 ts_tablet_manager.cc:1398] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:25.897377 17014 raft_consensus.cc:482] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.897506 17014 raft_consensus.cc:504] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:25.898061 17014 leader_election.cc:288] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:25.898411 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48de8e0c12b432ea5079390400302c1" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:25.898495 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48de8e0c12b432ea5079390400302c1" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:25.898614 16005 raft_consensus.cc:2454] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:25.898689 16265 raft_consensus.cc:2454] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:25.899008 15810 leader_election.cc:302] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:25.899258 17014 raft_consensus.cc:2781] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:25.899350 17014 raft_consensus.cc:482] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:25.899411 17014 raft_consensus.cc:3037] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.901105 17014 raft_consensus.cc:504] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:25.901563 17014 leader_election.cc:288] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:25.901918 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48de8e0c12b432ea5079390400302c1" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:25.901996 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48de8e0c12b432ea5079390400302c1" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:25.902096 16005 raft_consensus.cc:3037] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.902208 16265 raft_consensus.cc:3037] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.904752 16005 raft_consensus.cc:2454] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:25.904839 16265 raft_consensus.cc:2454] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:25.905138 15810 leader_election.cc:302] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:25.905418 17014 raft_consensus.cc:2781] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:25.905716 17014 raft_consensus.cc:686] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Running, Role: LEADER
I20240501 20:10:25.905969 17014 consensus_queue.cc:237] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:25.908236 15733 catalog_manager.cc:5484] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:25.918589 17014 raft_consensus.cc:482] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.918727 17014 raft_consensus.cc:504] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:25.919138 17014 leader_election.cc:288] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:25.919539 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fb9bc4cd73492da44bde1eecd627dc" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:25.919602 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7fb9bc4cd73492da44bde1eecd627dc" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:25.919870 16134 raft_consensus.cc:2379] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in current term 1: Already voted for candidate 754fcd3b82f041568f546c6205f44272 in this term.
I20240501 20:10:25.920224 15810 leader_election.cc:302] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272
I20240501 20:10:25.920459 17014 raft_consensus.cc:3037] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.922189 17014 raft_consensus.cc:2726] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:25.950884 17070 raft_consensus.cc:482] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.951048 17070 raft_consensus.cc:504] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:25.951597 17070 leader_election.cc:288] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:25.951996 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1474070bbdf147d2ae3313fdf9d71c19" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:25.952044 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1474070bbdf147d2ae3313fdf9d71c19" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:25.952174 15873 raft_consensus.cc:2454] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:25.952232 16134 raft_consensus.cc:2454] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:25.952554 16200 leader_election.cc:302] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:25.952816 17070 raft_consensus.cc:2781] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:25.952908 17070 raft_consensus.cc:482] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:25.953007 17070 raft_consensus.cc:3037] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.954802 17070 raft_consensus.cc:504] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:25.955329 17070 leader_election.cc:288] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Requested vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:25.955636 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1474070bbdf147d2ae3313fdf9d71c19" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:25.955713 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1474070bbdf147d2ae3313fdf9d71c19" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:25.955780 15873 raft_consensus.cc:3037] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.955896 16134 raft_consensus.cc:3037] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.957547 15873 raft_consensus.cc:2454] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:25.957701 16134 raft_consensus.cc:2454] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:25.957876 16200 leader_election.cc:302] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:25.958132 17070 raft_consensus.cc:2781] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:25.958317 17070 raft_consensus.cc:686] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Becoming Leader. State: Replica: f856e0d1074b431eb19763847958b89e, State: Running, Role: LEADER
I20240501 20:10:25.958542 17070 consensus_queue.cc:237] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:25.960983 15733 catalog_manager.cc:5484] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e reported cstate change: term changed from 0 to 1, leader changed from <none> to f856e0d1074b431eb19763847958b89e (127.8.68.196). New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:26.277745 17166 consensus_queue.cc:1035] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.285302 17166 consensus_queue.cc:1035] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.294340 17014 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:26.298130 17230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:26.298527 17230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:10:26.298856 17227 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:26.307724 17230 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 20:10:26.363502 17070 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.372022 17144 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 208.748 ms
  time per row: 0.104374 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:10:27.505826 15730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:27.514586 15986 tablet_service.cc:1467] Processing CreateTablet for tablet c538581c71f646b5bc8d71cb6981f201 (DEFAULT_TABLE table=rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.514837 16116 tablet_service.cc:1467] Processing CreateTablet for tablet c538581c71f646b5bc8d71cb6981f201 (DEFAULT_TABLE table=rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.515084 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c538581c71f646b5bc8d71cb6981f201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.515261 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c538581c71f646b5bc8d71cb6981f201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.516381 16115 tablet_service.cc:1467] Processing CreateTablet for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b (DEFAULT_TABLE table=rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.516610 16244 tablet_service.cc:1467] Processing CreateTablet for tablet f345303c72774848bb60e806cff22d12 (DEFAULT_TABLE table=rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.516779 16115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.516974 16244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f345303c72774848bb60e806cff22d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.518821 16114 tablet_service.cc:1467] Processing CreateTablet for tablet f345303c72774848bb60e806cff22d12 (DEFAULT_TABLE table=rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.519217 16114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f345303c72774848bb60e806cff22d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.522660 16245 tablet_service.cc:1467] Processing CreateTablet for tablet c538581c71f646b5bc8d71cb6981f201 (DEFAULT_TABLE table=rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.523038 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c538581c71f646b5bc8d71cb6981f201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.527220 15853 tablet_service.cc:1467] Processing CreateTablet for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b (DEFAULT_TABLE table=rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.527367 17295 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:27.527622 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.528673 17295 tablet_bootstrap.cc:654] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.531131 15984 tablet_service.cc:1467] Processing CreateTablet for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b (DEFAULT_TABLE table=rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.531522 15984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.532567 17295 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:27.532693 17295 ts_tablet_manager.cc:1367] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 20:10:27.533421 17295 raft_consensus.cc:348] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.533624 17295 raft_consensus.cc:374] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.533725 17295 raft_consensus.cc:729] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.533941 17295 consensus_queue.cc:260] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.536552 17295 ts_tablet_manager.cc:1398] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:10:27.536803 17295 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:27.536844 15985 tablet_service.cc:1467] Processing CreateTablet for tablet f345303c72774848bb60e806cff22d12 (DEFAULT_TABLE table=rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.537178 15985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f345303c72774848bb60e806cff22d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.538142 17295 tablet_bootstrap.cc:654] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.541960 17298 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:27.543948 17298 tablet_bootstrap.cc:654] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.545120 17296 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:27.546765 17296 tablet_bootstrap.cc:654] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.550614 17295 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:27.550822 17295 ts_tablet_manager.cc:1367] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.007s
I20240501 20:10:27.551000 17300 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:27.551507 17295 raft_consensus.cc:348] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.551744 17296 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:27.551786 17295 raft_consensus.cc:374] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.551921 17295 raft_consensus.cc:729] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.552189 17295 consensus_queue.cc:260] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.552356 17296 ts_tablet_manager.cc:1367] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.002s
I20240501 20:10:27.552819 17295 ts_tablet_manager.cc:1398] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:10:27.552865 17300 tablet_bootstrap.cc:654] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.553135 17295 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:27.553234 17296 raft_consensus.cc:348] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.554456 17296 raft_consensus.cc:374] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.554585 17296 raft_consensus.cc:729] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.555023 17295 tablet_bootstrap.cc:654] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.555004 17296 consensus_queue.cc:260] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.555779 17298 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:27.555963 17298 ts_tablet_manager.cc:1367] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.000s
I20240501 20:10:27.556732 17298 raft_consensus.cc:348] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.556996 17298 raft_consensus.cc:374] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.557147 17298 raft_consensus.cc:729] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.557426 17298 consensus_queue.cc:260] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.557948 17298 ts_tablet_manager.cc:1398] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:27.558607 17296 ts_tablet_manager.cc:1398] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.001s
I20240501 20:10:27.558910 17296 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:27.560376 17296 tablet_bootstrap.cc:654] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.560622 17279 raft_consensus.cc:482] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:27.560757 17279 raft_consensus.cc:504] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.561381 17279 leader_election.cc:288] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:27.562063 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a1a818ab7e44a5ac5c67f42e1dfe3b" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:27.562263 16134 raft_consensus.cc:2454] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:27.563004 15810 leader_election.cc:302] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 
I20240501 20:10:27.563009 16004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a1a818ab7e44a5ac5c67f42e1dfe3b" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:27.563345 17279 raft_consensus.cc:2781] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:27.563452 17279 raft_consensus.cc:482] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:27.563530 17279 raft_consensus.cc:3037] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.563938 17295 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:27.564056 17295 ts_tablet_manager.cc:1367] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.006s
I20240501 20:10:27.564708 17295 raft_consensus.cc:348] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.564890 17295 raft_consensus.cc:374] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.564972 17295 raft_consensus.cc:729] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.565174 17295 consensus_queue.cc:260] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.565364 17300 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:27.565485 17300 ts_tablet_manager.cc:1367] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.007s
I20240501 20:10:27.565670 17295 ts_tablet_manager.cc:1398] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:10:27.565790 17279 raft_consensus.cc:504] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.566133 17300 raft_consensus.cc:348] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.566306 17300 raft_consensus.cc:374] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.566416 17300 raft_consensus.cc:729] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.566619 17300 consensus_queue.cc:260] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.567104 17300 ts_tablet_manager.cc:1398] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
W20240501 20:10:27.567253 15810 leader_election.cc:341] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:27.567399 17300 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:27.567675 17279 leader_election.cc:288] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:27.568739 17300 tablet_bootstrap.cc:654] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.569485 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a1a818ab7e44a5ac5c67f42e1dfe3b" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:27.569690 16134 raft_consensus.cc:3037] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.570364 16004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a1a818ab7e44a5ac5c67f42e1dfe3b" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
W20240501 20:10:27.570919 15810 leader_election.cc:341] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:27.575408 17296 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:27.575599 17296 ts_tablet_manager.cc:1367] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.004s
I20240501 20:10:27.575994 16134 raft_consensus.cc:2454] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:27.576381 15810 leader_election.cc:302] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 754fcd3b82f041568f546c6205f44272
I20240501 20:10:27.576653 17279 raft_consensus.cc:2781] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:27.576802 17279 raft_consensus.cc:686] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Running, Role: LEADER
I20240501 20:10:27.577061 17279 consensus_queue.cc:237] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.576381 17296 raft_consensus.cc:348] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.580937 17296 raft_consensus.cc:374] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.581027 17296 raft_consensus.cc:729] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.581238 17296 consensus_queue.cc:260] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.582302 17296 ts_tablet_manager.cc:1398] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.001s
I20240501 20:10:27.585670 17300 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:27.585855 17300 ts_tablet_manager.cc:1367] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.018s	user 0.007s	sys 0.009s
I20240501 20:10:27.586555 17300 raft_consensus.cc:348] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.586807 17300 raft_consensus.cc:374] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.586899 17300 raft_consensus.cc:729] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.587169 17300 consensus_queue.cc:260] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.588373 17300 ts_tablet_manager.cc:1398] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:27.588718 17300 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:27.588909 15733 catalog_manager.cc:5484] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:27.590209 17300 tablet_bootstrap.cc:654] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.593257 17300 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:27.593422 17300 ts_tablet_manager.cc:1367] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240501 20:10:27.594113 17300 raft_consensus.cc:348] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.594264 17300 raft_consensus.cc:374] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.594396 17300 raft_consensus.cc:729] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.594588 17300 consensus_queue.cc:260] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.594995 17300 ts_tablet_manager.cc:1398] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240501 20:10:27.664261 16970 raft_consensus.cc:482] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:27.664419 16970 raft_consensus.cc:504] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.665088 16970 leader_election.cc:288] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:27.665403 16004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345303c72774848bb60e806cff22d12" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:27.665513 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345303c72774848bb60e806cff22d12" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:27.665594 16004 raft_consensus.cc:2454] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:27.665714 16266 raft_consensus.cc:2454] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:27.665944 16071 leader_election.cc:302] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:27.666216 16970 raft_consensus.cc:2781] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:27.666339 16970 raft_consensus.cc:482] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:27.666424 16970 raft_consensus.cc:3037] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.668200 16970 raft_consensus.cc:504] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.668668 16970 leader_election.cc:288] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:27.669046 16004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345303c72774848bb60e806cff22d12" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:27.669144 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f345303c72774848bb60e806cff22d12" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:27.669234 16004 raft_consensus.cc:3037] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.669317 16266 raft_consensus.cc:3037] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.671880 16004 raft_consensus.cc:2454] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:27.671891 16266 raft_consensus.cc:2454] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:27.672241 16069 leader_election.cc:302] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:27.672500 16970 raft_consensus.cc:2781] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:27.672885 16970 raft_consensus.cc:686] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Becoming Leader. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Running, Role: LEADER
I20240501 20:10:27.673267 16970 consensus_queue.cc:237] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.675822 15733 catalog_manager.cc:5484] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:27.708983 17070 raft_consensus.cc:482] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:27.709158 17070 raft_consensus.cc:504] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.709762 17070 leader_election.cc:288] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:27.710768 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c538581c71f646b5bc8d71cb6981f201" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:27.710992 16134 raft_consensus.cc:2454] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:27.711364 16201 leader_election.cc:302] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:27.711607 17070 raft_consensus.cc:2781] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:27.711690 17070 raft_consensus.cc:482] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:27.711792 17070 raft_consensus.cc:3037] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.712750 17188 raft_consensus.cc:482] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:27.712885 17188 raft_consensus.cc:504] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:27.713513 17188 leader_election.cc:288] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:27.713868 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a1a818ab7e44a5ac5c67f42e1dfe3b" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:27.714059 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a1a818ab7e44a5ac5c67f42e1dfe3b" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:27.714068 17070 raft_consensus.cc:504] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.714301 16134 raft_consensus.cc:2379] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 754fcd3b82f041568f546c6205f44272 in current term 1: Already voted for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in this term.
I20240501 20:10:27.714720 17070 leader_election.cc:288] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:27.714766 15941 leader_election.cc:302] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fcd3b82f041568f546c6205f44272; no voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2
I20240501 20:10:27.715118 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c538581c71f646b5bc8d71cb6981f201" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:27.715284 17188 raft_consensus.cc:3037] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.715317 16134 raft_consensus.cc:3037] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.716604 16004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c538581c71f646b5bc8d71cb6981f201" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:27.716771 16004 raft_consensus.cc:2454] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:27.717859 16134 raft_consensus.cc:2454] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:27.717993 17188 raft_consensus.cc:2726] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:27.718189 16201 leader_election.cc:302] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:27.718185 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c538581c71f646b5bc8d71cb6981f201" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:27.718358 16005 raft_consensus.cc:3037] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.718734 17313 raft_consensus.cc:2781] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:27.719087 17313 raft_consensus.cc:686] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Becoming Leader. State: Replica: f856e0d1074b431eb19763847958b89e, State: Running, Role: LEADER
I20240501 20:10:27.719563 17313 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:27.720737 16005 raft_consensus.cc:2454] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:27.722244 15733 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e reported cstate change: term changed from 0 to 1, leader changed from <none> to f856e0d1074b431eb19763847958b89e (127.8.68.196). New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:27.995553 17279 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:28.003337 17279 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:28.094061 17316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:28.094352 17316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:28.101433 17316 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:28.113287 16970 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:28.117204 17309 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240501 20:10:28.169961 17070 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:28.183205 17314 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 204.716 ms
  time per row: 0.102358 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:10:29.308684 15733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:29.317071 15986 tablet_service.cc:1467] Processing CreateTablet for tablet b016fc1d28524a02bd762b74c6b08375 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:29.317253 16244 tablet_service.cc:1467] Processing CreateTablet for tablet b016fc1d28524a02bd762b74c6b08375 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:29.317523 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016fc1d28524a02bd762b74c6b08375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.317679 16244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016fc1d28524a02bd762b74c6b08375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.319033 15854 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6b15e4aebf4b1f9a3176f87083703e (DEFAULT_TABLE table=rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:29.319231 15853 tablet_service.cc:1467] Processing CreateTablet for tablet b016fc1d28524a02bd762b74c6b08375 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:29.319453 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6b15e4aebf4b1f9a3176f87083703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.319684 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016fc1d28524a02bd762b74c6b08375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.322712 16114 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6b15e4aebf4b1f9a3176f87083703e (DEFAULT_TABLE table=rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:29.323181 16114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6b15e4aebf4b1f9a3176f87083703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.323458 15855 tablet_service.cc:1467] Processing CreateTablet for tablet ddf5ddac08364043bf2e27184d5d3213 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:29.323793 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf5ddac08364043bf2e27184d5d3213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.326917 15984 tablet_service.cc:1467] Processing CreateTablet for tablet ddf5ddac08364043bf2e27184d5d3213 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:29.327358 15984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf5ddac08364043bf2e27184d5d3213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.328076 16245 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6b15e4aebf4b1f9a3176f87083703e (DEFAULT_TABLE table=rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:29.328456 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6b15e4aebf4b1f9a3176f87083703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.330631 16246 tablet_service.cc:1467] Processing CreateTablet for tablet ddf5ddac08364043bf2e27184d5d3213 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:29.330961 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf5ddac08364043bf2e27184d5d3213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.332931 17382 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:29.336452 17382 tablet_bootstrap.cc:654] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:29.337996 17383 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:29.338752 17382 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:29.338920 17382 ts_tablet_manager.cc:1367] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240501 20:10:29.339460 17382 raft_consensus.cc:348] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.339622 17382 raft_consensus.cc:374] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:29.339741 17382 raft_consensus.cc:729] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.339958 17382 consensus_queue.cc:260] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.340457 17382 ts_tablet_manager.cc:1398] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240501 20:10:29.344049 17382 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:29.345156 17383 tablet_bootstrap.cc:654] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:29.345516 17382 tablet_bootstrap.cc:654] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:29.345795 17386 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:29.347481 17386 tablet_bootstrap.cc:654] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:29.349086 17384 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:29.351063 17384 tablet_bootstrap.cc:654] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:29.354825 17382 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:29.355043 17382 ts_tablet_manager.cc:1367] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.002s
I20240501 20:10:29.355801 17382 raft_consensus.cc:348] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.356089 17382 raft_consensus.cc:374] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:29.356205 17382 raft_consensus.cc:729] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.356514 17382 consensus_queue.cc:260] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.357945 17382 ts_tablet_manager.cc:1398] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 20:10:29.358273 17382 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:29.358546 17386 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:29.358727 17386 ts_tablet_manager.cc:1367] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.005s
I20240501 20:10:29.359696 17382 tablet_bootstrap.cc:654] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:29.360618 17386 raft_consensus.cc:348] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.360870 17386 raft_consensus.cc:374] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:29.361002 17386 raft_consensus.cc:729] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.361403 17386 consensus_queue.cc:260] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.361694 17279 raft_consensus.cc:482] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:29.361832 17279 raft_consensus.cc:504] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.362470 17279 leader_election.cc:288] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:29.363118 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5ddac08364043bf2e27184d5d3213" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:29.363332 16266 raft_consensus.cc:2454] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:29.363752 15808 leader_election.cc:302] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:29.364538 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5ddac08364043bf2e27184d5d3213" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:29.364841 17279 raft_consensus.cc:2781] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:29.364956 17279 raft_consensus.cc:482] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:29.365090 17279 raft_consensus.cc:3037] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:10:29.365171 15810 leader_election.cc:341] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:29.365839 17384 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:29.365962 17384 ts_tablet_manager.cc:1367] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.008s
I20240501 20:10:29.366698 17384 raft_consensus.cc:348] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:29.366876 17384 raft_consensus.cc:374] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:29.366947 17384 raft_consensus.cc:729] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.367130 17384 consensus_queue.cc:260] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:29.367583 17279 raft_consensus.cc:504] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.368208 17382 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:29.368369 17382 ts_tablet_manager.cc:1367] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.003s
I20240501 20:10:29.368525 17279 leader_election.cc:288] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:29.369014 17386 ts_tablet_manager.cc:1398] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.001s
I20240501 20:10:29.369135 17382 raft_consensus.cc:348] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:29.369158 17314 raft_consensus.cc:482] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:29.369343 17382 raft_consensus.cc:374] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:29.369428 17382 raft_consensus.cc:729] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.369566 17314 raft_consensus.cc:504] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.369632 17382 consensus_queue.cc:260] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:29.370267 17386 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:29.371883 17386 tablet_bootstrap.cc:654] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:29.372387 17384 ts_tablet_manager.cc:1398] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 20:10:29.372505 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016fc1d28524a02bd762b74c6b08375" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:29.373437 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5ddac08364043bf2e27184d5d3213" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:29.373642 16266 raft_consensus.cc:3037] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:29.375716 17383 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:29.375856 17383 ts_tablet_manager.cc:1367] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.038s	user 0.011s	sys 0.000s
I20240501 20:10:29.376302 16266 raft_consensus.cc:2454] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:29.376708 15808 leader_election.cc:302] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:29.376981 17279 raft_consensus.cc:2781] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:29.377051 17383 raft_consensus.cc:348] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.377274 17383 raft_consensus.cc:374] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:29.377360 17383 raft_consensus.cc:729] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.370237 17314 leader_election.cc:288] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:29.377669 17383 consensus_queue.cc:260] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.377663 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016fc1d28524a02bd762b74c6b08375" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:29.378054 16005 raft_consensus.cc:2454] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:29.378434 17383 ts_tablet_manager.cc:1398] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:10:29.378692 17383 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:29.378795 16201 leader_election.cc:302] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754fcd3b82f041568f546c6205f44272, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:29.380738 17383 tablet_bootstrap.cc:654] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:29.384095 17314 raft_consensus.cc:2781] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:29.384198 17314 raft_consensus.cc:482] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:29.384296 17314 raft_consensus.cc:3037] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:10:29.389000 16200 leader_election.cc:341] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:29.390388 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5ddac08364043bf2e27184d5d3213" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:29.390600 17314 raft_consensus.cc:504] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
W20240501 20:10:29.391053 15810 leader_election.cc:341] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:29.391523 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016fc1d28524a02bd762b74c6b08375" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:29.391714 16005 raft_consensus.cc:3037] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:29.392971 17314 leader_election.cc:288] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:29.393550 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016fc1d28524a02bd762b74c6b08375" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
W20240501 20:10:29.396222 16200 leader_election.cc:341] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:29.398029 16005 raft_consensus.cc:2454] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:29.398425 16201 leader_election.cc:302] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 754fcd3b82f041568f546c6205f44272, f856e0d1074b431eb19763847958b89e; no voters: 51b81a5ae56b4dd99af1b0167c1da1e2
I20240501 20:10:29.398931 17314 raft_consensus.cc:2781] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:29.399144 17314 raft_consensus.cc:686] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Becoming Leader. State: Replica: f856e0d1074b431eb19763847958b89e, State: Running, Role: LEADER
I20240501 20:10:29.399426 17314 consensus_queue.cc:237] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.400167 17279 raft_consensus.cc:686] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Running, Role: LEADER
I20240501 20:10:29.400496 17279 consensus_queue.cc:237] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.401150 17383 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:29.401311 17383 ts_tablet_manager.cc:1367] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.004s
I20240501 20:10:29.401990 17383 raft_consensus.cc:348] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.402204 17383 raft_consensus.cc:374] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:29.402295 17383 raft_consensus.cc:729] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.402510 17383 consensus_queue.cc:260] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.405321 17382 ts_tablet_manager.cc:1398] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.037s	user 0.018s	sys 0.013s
I20240501 20:10:29.405937 17383 ts_tablet_manager.cc:1398] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240501 20:10:29.406157 17386 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:29.406347 17386 ts_tablet_manager.cc:1367] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.021s
I20240501 20:10:29.406543 15730 catalog_manager.cc:5484] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:29.406723 15733 catalog_manager.cc:5484] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e reported cstate change: term changed from 0 to 1, leader changed from <none> to f856e0d1074b431eb19763847958b89e (127.8.68.196). New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:29.407135 17386 raft_consensus.cc:348] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.407322 17386 raft_consensus.cc:374] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:29.407388 17386 raft_consensus.cc:729] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.407733 17386 consensus_queue.cc:260] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.408278 17386 ts_tablet_manager.cc:1398] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:10:29.408587 17386 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:29.409978 17386 tablet_bootstrap.cc:654] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:29.412426 17386 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:29.412619 17386 ts_tablet_manager.cc:1367] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:10:29.413229 17386 raft_consensus.cc:348] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:29.413358 17386 raft_consensus.cc:374] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:29.413420 17386 raft_consensus.cc:729] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:29.413579 17386 consensus_queue.cc:260] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:29.413941 17386 ts_tablet_manager.cc:1398] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:10:29.501559 17279 raft_consensus.cc:482] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:29.501688 17279 raft_consensus.cc:504] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:29.502152 17279 leader_election.cc:288] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:29.502504 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b15e4aebf4b1f9a3176f87083703e" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:29.502562 16135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b15e4aebf4b1f9a3176f87083703e" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:29.502671 16266 raft_consensus.cc:2454] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:29.502782 16135 raft_consensus.cc:2454] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 0.
I20240501 20:10:29.503031 15808 leader_election.cc:302] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:29.503240 17279 raft_consensus.cc:2781] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:29.503332 17279 raft_consensus.cc:482] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:29.503401 17279 raft_consensus.cc:3037] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:29.505097 17279 raft_consensus.cc:504] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:29.505546 17279 leader_election.cc:288] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Requested vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:29.505893 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b15e4aebf4b1f9a3176f87083703e" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:29.505952 16135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b15e4aebf4b1f9a3176f87083703e" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:29.506081 16266 raft_consensus.cc:3037] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:29.506121 16135 raft_consensus.cc:3037] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:29.508736 16266 raft_consensus.cc:2454] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:29.508736 16135 raft_consensus.cc:2454] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in term 1.
I20240501 20:10:29.509130 15808 leader_election.cc:302] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:29.509409 17279 raft_consensus.cc:2781] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:29.509538 17279 raft_consensus.cc:686] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Running, Role: LEADER
I20240501 20:10:29.509755 17279 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:29.512215 15731 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:29.519903 17279 raft_consensus.cc:482] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:29.520025 17279 raft_consensus.cc:504] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.520494 17279 leader_election.cc:288] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:29.520866 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016fc1d28524a02bd762b74c6b08375" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:29.520910 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b016fc1d28524a02bd762b74c6b08375" candidate_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:29.521153 16005 raft_consensus.cc:2379] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in current term 1: Already voted for candidate f856e0d1074b431eb19763847958b89e in this term.
I20240501 20:10:29.521528 15810 leader_election.cc:302] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 754fcd3b82f041568f546c6205f44272, f856e0d1074b431eb19763847958b89e
I20240501 20:10:29.521770 17279 raft_consensus.cc:3037] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:29.523649 17279 raft_consensus.cc:2726] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:29.765051 17188 raft_consensus.cc:482] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:29.765257 17188 raft_consensus.cc:504] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:29.765934 17188 leader_election.cc:288] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:29.766338 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5ddac08364043bf2e27184d5d3213" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:29.766455 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf5ddac08364043bf2e27184d5d3213" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:29.766635 16266 raft_consensus.cc:2379] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 754fcd3b82f041568f546c6205f44272 in current term 1: Already voted for candidate 51b81a5ae56b4dd99af1b0167c1da1e2 in this term.
I20240501 20:10:29.767067 15939 leader_election.cc:302] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fcd3b82f041568f546c6205f44272; no voters: 51b81a5ae56b4dd99af1b0167c1da1e2, f856e0d1074b431eb19763847958b89e
I20240501 20:10:29.767438 17188 raft_consensus.cc:3037] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:29.770550 17188 raft_consensus.cc:2726] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:29.786146 17279 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.807883 17279 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.810140 17314 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.823433 17394 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:29.872748 17403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:29.873049 17403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:29.880425 17403 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:29.899192 17408 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.904470 17409 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'

INSERT report
    rows total: 2000
    time total: 193.106 ms
  time per row: 0.0965532 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      6       1788 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:10:31.074412 15728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:31.082669 16114 tablet_service.cc:1467] Processing CreateTablet for tablet ff11a8e00b544b4193a2824541668492 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:31.083132 16114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff11a8e00b544b4193a2824541668492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.083230 15855 tablet_service.cc:1467] Processing CreateTablet for tablet ff11a8e00b544b4193a2824541668492 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:31.083603 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff11a8e00b544b4193a2824541668492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.083902 15986 tablet_service.cc:1467] Processing CreateTablet for tablet ff11a8e00b544b4193a2824541668492 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:31.084232 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff11a8e00b544b4193a2824541668492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.086331 16245 tablet_service.cc:1467] Processing CreateTablet for tablet eb90224d08f84a2faf7dcc063c6af56e (DEFAULT_TABLE table=rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:31.086736 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb90224d08f84a2faf7dcc063c6af56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.087015 16116 tablet_service.cc:1467] Processing CreateTablet for tablet eb90224d08f84a2faf7dcc063c6af56e (DEFAULT_TABLE table=rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:31.087047 15984 tablet_service.cc:1467] Processing CreateTablet for tablet d3ba52a871714255aa1bd34d9daca920 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:31.087352 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb90224d08f84a2faf7dcc063c6af56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.087385 15984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ba52a871714255aa1bd34d9daca920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.090901 16246 tablet_service.cc:1467] Processing CreateTablet for tablet d3ba52a871714255aa1bd34d9daca920 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:31.091233 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ba52a871714255aa1bd34d9daca920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.094617 16115 tablet_service.cc:1467] Processing CreateTablet for tablet d3ba52a871714255aa1bd34d9daca920 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:31.094944 16115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ba52a871714255aa1bd34d9daca920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.100677 15854 tablet_service.cc:1467] Processing CreateTablet for tablet eb90224d08f84a2faf7dcc063c6af56e (DEFAULT_TABLE table=rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:31.101081 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb90224d08f84a2faf7dcc063c6af56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.107236 17467 tablet_bootstrap.cc:492] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:31.109535 17467 tablet_bootstrap.cc:654] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:31.109647 17468 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:31.110680 17465 tablet_bootstrap.cc:492] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:31.111351 17468 tablet_bootstrap.cc:654] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:31.111876 17465 tablet_bootstrap.cc:654] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:31.112720 17466 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:31.113719 17467 tablet_bootstrap.cc:492] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:31.113871 17467 ts_tablet_manager.cc:1367] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240501 20:10:31.114162 17466 tablet_bootstrap.cc:654] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:31.114638 17467 raft_consensus.cc:348] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.114948 17467 raft_consensus.cc:374] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:31.115064 17467 raft_consensus.cc:729] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.115372 17468 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:31.115357 17467 consensus_queue.cc:260] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.115906 17467 ts_tablet_manager.cc:1398] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:31.116331 17467 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:31.117183 17468 ts_tablet_manager.cc:1367] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240501 20:10:31.117208 17465 tablet_bootstrap.cc:492] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:31.117326 17465 ts_tablet_manager.cc:1367] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.005s
I20240501 20:10:31.118041 17467 tablet_bootstrap.cc:654] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:31.117941 17468 raft_consensus.cc:348] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.117978 17465 raft_consensus.cc:348] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.118130 17468 raft_consensus.cc:374] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:31.118151 17465 raft_consensus.cc:374] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:31.118211 17468 raft_consensus.cc:729] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.118229 17465 raft_consensus.cc:729] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.118446 17468 consensus_queue.cc:260] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.118492 17465 consensus_queue.cc:260] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.118964 17468 ts_tablet_manager.cc:1398] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:31.118984 17465 ts_tablet_manager.cc:1398] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:31.119218 17468 tablet_bootstrap.cc:492] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:31.119212 17465 tablet_bootstrap.cc:492] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:31.120635 17468 tablet_bootstrap.cc:654] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:31.122859 17465 tablet_bootstrap.cc:654] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:31.128439 17466 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:31.128633 17466 ts_tablet_manager.cc:1367] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.000s
I20240501 20:10:31.129514 17466 raft_consensus.cc:348] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.129746 17466 raft_consensus.cc:374] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:31.129864 17466 raft_consensus.cc:729] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.130085 17466 consensus_queue.cc:260] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.136667 17467 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:31.136885 17467 ts_tablet_manager.cc:1367] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.021s	user 0.000s	sys 0.015s
I20240501 20:10:31.137746 17467 raft_consensus.cc:348] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.138029 17467 raft_consensus.cc:374] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:31.138154 17467 raft_consensus.cc:729] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.138643 17314 raft_consensus.cc:482] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:31.138825 17314 raft_consensus.cc:504] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.139454 17314 leader_election.cc:288] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:31.140269 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb90224d08f84a2faf7dcc063c6af56e" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:31.141291 16135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb90224d08f84a2faf7dcc063c6af56e" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:31.141503 16135 raft_consensus.cc:2454] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:31.142058 16201 leader_election.cc:302] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:31.142526 17314 raft_consensus.cc:2781] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:31.142623 17314 raft_consensus.cc:482] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:31.142729 17314 raft_consensus.cc:3037] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:31.138566 17467 consensus_queue.cc:260] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.145013 15873 raft_consensus.cc:2454] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:31.145484 17314 raft_consensus.cc:504] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.146469 17467 ts_tablet_manager.cc:1398] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240501 20:10:31.147397 16135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb90224d08f84a2faf7dcc063c6af56e" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:31.147617 16135 raft_consensus.cc:3037] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:31.150624 16135 raft_consensus.cc:2454] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:31.151666 16201 leader_election.cc:302] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:31.151953 17476 raft_consensus.cc:2781] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:31.156387 17314 leader_election.cc:288] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Requested vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:31.156988 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb90224d08f84a2faf7dcc063c6af56e" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:31.157187 15873 raft_consensus.cc:3037] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:31.159751 15873 raft_consensus.cc:2454] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:31.162189 17466 ts_tablet_manager.cc:1398] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.033s	user 0.002s	sys 0.000s
I20240501 20:10:31.164391 17476 raft_consensus.cc:686] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Becoming Leader. State: Replica: f856e0d1074b431eb19763847958b89e, State: Running, Role: LEADER
I20240501 20:10:31.165167 17466 tablet_bootstrap.cc:492] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:31.164777 17476 consensus_queue.cc:237] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.166788 17466 tablet_bootstrap.cc:654] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:31.172195 17468 tablet_bootstrap.cc:492] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:31.172478 17468 ts_tablet_manager.cc:1367] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.053s	user 0.031s	sys 0.018s
I20240501 20:10:31.173314 17468 raft_consensus.cc:348] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:31.173606 17468 raft_consensus.cc:374] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:31.173728 17468 raft_consensus.cc:729] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.174036 17468 consensus_queue.cc:260] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:31.178673 17468 ts_tablet_manager.cc:1398] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.000s
I20240501 20:10:31.179361 15728 catalog_manager.cc:5484] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e reported cstate change: term changed from 0 to 1, leader changed from <none> to f856e0d1074b431eb19763847958b89e (127.8.68.196). New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:31.181742 17466 tablet_bootstrap.cc:492] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:31.181943 17466 ts_tablet_manager.cc:1367] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.017s	user 0.003s	sys 0.000s
I20240501 20:10:31.183869 17466 raft_consensus.cc:348] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.184144 17466 raft_consensus.cc:374] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:31.184218 17466 raft_consensus.cc:729] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.184453 17466 consensus_queue.cc:260] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.184659 17465 tablet_bootstrap.cc:492] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:31.184885 17465 ts_tablet_manager.cc:1367] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.066s	user 0.012s	sys 0.033s
I20240501 20:10:31.185206 17466 ts_tablet_manager.cc:1398] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:10:31.185503 17466 tablet_bootstrap.cc:492] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:31.185632 17465 raft_consensus.cc:348] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:31.185874 17465 raft_consensus.cc:374] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:31.185976 17465 raft_consensus.cc:729] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.186229 17465 consensus_queue.cc:260] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:31.186839 17465 ts_tablet_manager.cc:1398] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:31.186916 17466 tablet_bootstrap.cc:654] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:31.188807 17466 tablet_bootstrap.cc:492] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:31.188937 17466 ts_tablet_manager.cc:1367] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:10:31.189514 17466 raft_consensus.cc:348] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:31.189663 17466 raft_consensus.cc:374] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:31.189778 17466 raft_consensus.cc:729] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:31.190052 17466 consensus_queue.cc:260] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:31.190690 17466 ts_tablet_manager.cc:1398] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:10:31.265714 16970 raft_consensus.cc:482] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:31.265847 16970 raft_consensus.cc:504] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.266546 16970 leader_election.cc:288] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:31.266836 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff11a8e00b544b4193a2824541668492" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:31.267083 16005 raft_consensus.cc:2454] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:31.267450 16071 leader_election.cc:302] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:31.267709 16970 raft_consensus.cc:2781] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:31.267849 16970 raft_consensus.cc:482] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:31.267959 16970 raft_consensus.cc:3037] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:31.270442 16970 raft_consensus.cc:504] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.270967 16970 leader_election.cc:288] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:31.271272 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff11a8e00b544b4193a2824541668492" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:31.271425 16005 raft_consensus.cc:3037] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:31.273653 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff11a8e00b544b4193a2824541668492" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:31.273638 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff11a8e00b544b4193a2824541668492" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:31.273769 16005 raft_consensus.cc:2454] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:31.273881 15875 raft_consensus.cc:3037] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:31.274111 16071 leader_election.cc:302] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:31.274426 16970 raft_consensus.cc:2781] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:31.274780 16970 raft_consensus.cc:686] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Becoming Leader. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Running, Role: LEADER
I20240501 20:10:31.275086 16970 consensus_queue.cc:237] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:31.276612 15875 raft_consensus.cc:2454] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:31.278180 15730 catalog_manager.cc:5484] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:31.543258 17476 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:31.547549 17476 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:31.627988 17476 raft_consensus.cc:482] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:31.628155 17476 raft_consensus.cc:504] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:31.628641 17476 leader_election.cc:288] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:31.628988 16135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ba52a871714255aa1bd34d9daca920" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:31.629048 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ba52a871714255aa1bd34d9daca920" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:31.629196 16135 raft_consensus.cc:2454] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:31.629282 16005 raft_consensus.cc:2454] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:31.629554 16201 leader_election.cc:302] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:31.629793 17476 raft_consensus.cc:2781] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:31.629876 17476 raft_consensus.cc:482] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:31.629966 17476 raft_consensus.cc:3037] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:31.631776 17476 raft_consensus.cc:504] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:31.632236 17476 leader_election.cc:288] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Requested vote from peers 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:31.632589 16135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ba52a871714255aa1bd34d9daca920" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:31.632644 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ba52a871714255aa1bd34d9daca920" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:31.632740 16135 raft_consensus.cc:3037] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:31.632841 16005 raft_consensus.cc:3037] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:31.634598 16135 raft_consensus.cc:2454] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:31.634846 16005 raft_consensus.cc:2454] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:31.634949 16201 leader_election.cc:302] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:31.635201 17476 raft_consensus.cc:2781] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:31.635360 17476 raft_consensus.cc:686] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Becoming Leader. State: Replica: f856e0d1074b431eb19763847958b89e, State: Running, Role: LEADER
I20240501 20:10:31.635605 17476 consensus_queue.cc:237] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:31.638142 15728 catalog_manager.cc:5484] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e reported cstate change: term changed from 0 to 1, leader changed from <none> to f856e0d1074b431eb19763847958b89e (127.8.68.196). New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:31.755820 16970 consensus_queue.cc:1035] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:31.761426 17484 consensus_queue.cc:1035] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:31.978111 17492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:31.978413 17492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:31.985437 17492 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:32.013520 17476 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240501 20:10:32.024411 17314 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 232.842 ms
  time per row: 0.116421 ms
I20240501 20:10:32.373540 15728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:32.381919 15854 tablet_service.cc:1467] Processing CreateTablet for tablet 67d59d1654ef4ab3ac996f55a73b313f (DEFAULT_TABLE table=rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:32.382398 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d59d1654ef4ab3ac996f55a73b313f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.382387 15986 tablet_service.cc:1467] Processing CreateTablet for tablet 67d59d1654ef4ab3ac996f55a73b313f (DEFAULT_TABLE table=rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:32.382436 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 67d59d1654ef4ab3ac996f55a73b313f (DEFAULT_TABLE table=rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:32.382817 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d59d1654ef4ab3ac996f55a73b313f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.382891 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d59d1654ef4ab3ac996f55a73b313f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.384286 16246 tablet_service.cc:1467] Processing CreateTablet for tablet 36465c82164a4765a63590cfd5d413a4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:32.384694 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36465c82164a4765a63590cfd5d413a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.387032 15855 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0746bed2cf43b9bc7765c270a8aa62 (DEFAULT_TABLE table=rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:32.387406 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0746bed2cf43b9bc7765c270a8aa62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.388752 16245 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0746bed2cf43b9bc7765c270a8aa62 (DEFAULT_TABLE table=rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:32.389125 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0746bed2cf43b9bc7765c270a8aa62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.390709 15853 tablet_service.cc:1467] Processing CreateTablet for tablet 36465c82164a4765a63590cfd5d413a4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:32.391050 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36465c82164a4765a63590cfd5d413a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.395252 16115 tablet_service.cc:1467] Processing CreateTablet for tablet 36465c82164a4765a63590cfd5d413a4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:32.395342 15984 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0746bed2cf43b9bc7765c270a8aa62 (DEFAULT_TABLE table=rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:32.395644 16115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36465c82164a4765a63590cfd5d413a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.395725 15984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0746bed2cf43b9bc7765c270a8aa62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.400406 17552 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:32.402093 17552 tablet_bootstrap.cc:654] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:32.404371 17551 tablet_bootstrap.cc:492] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:32.405737 17551 tablet_bootstrap.cc:654] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:32.407243 17552 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:32.407378 17552 ts_tablet_manager.cc:1367] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240501 20:10:32.407655 17551 tablet_bootstrap.cc:492] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:32.407969 17551 ts_tablet_manager.cc:1367] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 20:10:32.408905 17551 raft_consensus.cc:348] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:32.408955 17552 raft_consensus.cc:348] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:32.409152 17551 raft_consensus.cc:374] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:32.409255 17551 raft_consensus.cc:729] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:32.409461 17551 consensus_queue.cc:260] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:32.409612 17552 raft_consensus.cc:374] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:32.409704 17552 raft_consensus.cc:729] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:32.410097 17551 ts_tablet_manager.cc:1398] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:10:32.410226 17552 consensus_queue.cc:260] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:32.410425 17551 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:32.411253 17552 ts_tablet_manager.cc:1398] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:10:32.411520 17552 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:32.411692 17551 tablet_bootstrap.cc:654] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:32.416455 17552 tablet_bootstrap.cc:654] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:32.418834 17556 tablet_bootstrap.cc:492] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:32.420797 17556 tablet_bootstrap.cc:654] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:32.422183 17550 tablet_bootstrap.cc:492] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:32.424000 17550 tablet_bootstrap.cc:654] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:32.425875 17552 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:32.426016 17552 ts_tablet_manager.cc:1367] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.004s
I20240501 20:10:32.426476 17551 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:32.426633 17551 ts_tablet_manager.cc:1367] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.009s
I20240501 20:10:32.426730 17552 raft_consensus.cc:348] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:32.426879 17550 tablet_bootstrap.cc:492] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:32.426926 17552 raft_consensus.cc:374] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:32.426993 17550 ts_tablet_manager.cc:1367] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 20:10:32.426997 17552 raft_consensus.cc:729] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:32.427242 17552 consensus_queue.cc:260] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:32.427325 17551 raft_consensus.cc:348] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:32.427529 17551 raft_consensus.cc:374] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:32.427598 17551 raft_consensus.cc:729] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:32.427722 17550 raft_consensus.cc:348] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:32.427886 17552 ts_tablet_manager.cc:1398] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:32.427892 17550 raft_consensus.cc:374] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:32.427809 17551 consensus_queue.cc:260] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:32.427970 17550 raft_consensus.cc:729] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:32.428398 17551 ts_tablet_manager.cc:1398] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:32.428326 17550 consensus_queue.cc:260] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:32.428866 17550 ts_tablet_manager.cc:1398] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:32.429391 17556 tablet_bootstrap.cc:492] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:32.429529 17556 ts_tablet_manager.cc:1367] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240501 20:10:32.429602 17550 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:32.430243 17556 raft_consensus.cc:348] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:32.430461 17556 raft_consensus.cc:374] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:32.430548 17556 raft_consensus.cc:729] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:32.430739 17556 consensus_queue.cc:260] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:32.431044 17550 tablet_bootstrap.cc:654] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:32.431870 17556 ts_tablet_manager.cc:1398] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:32.432134 17556 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:32.434393 17556 tablet_bootstrap.cc:654] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:32.437918 17550 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:32.438089 17550 ts_tablet_manager.cc:1367] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.007s
I20240501 20:10:32.438742 17550 raft_consensus.cc:348] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:32.438947 17550 raft_consensus.cc:374] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:32.439078 17550 raft_consensus.cc:729] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:32.439210 17556 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:32.439352 17556 ts_tablet_manager.cc:1367] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.006s
I20240501 20:10:32.439323 17550 consensus_queue.cc:260] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:32.439823 17550 ts_tablet_manager.cc:1398] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:10:32.440114 17550 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:32.440047 17556 raft_consensus.cc:348] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:32.440236 17556 raft_consensus.cc:374] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:32.440315 17556 raft_consensus.cc:729] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:32.440548 17556 consensus_queue.cc:260] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:32.441136 17556 ts_tablet_manager.cc:1398] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:10:32.441488 17550 tablet_bootstrap.cc:654] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:32.443603 17550 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:32.443734 17550 ts_tablet_manager.cc:1367] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.001s
I20240501 20:10:32.445384 17550 raft_consensus.cc:348] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:32.445515 17550 raft_consensus.cc:374] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:32.445596 17550 raft_consensus.cc:729] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:32.445765 17550 consensus_queue.cc:260] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:32.446136 17550 ts_tablet_manager.cc:1398] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:10:32.490898 17544 raft_consensus.cc:482] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:32.491097 17544 raft_consensus.cc:504] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:32.491786 17544 leader_election.cc:288] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:32.492151 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0746bed2cf43b9bc7765c270a8aa62" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:32.492228 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0746bed2cf43b9bc7765c270a8aa62" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:32.492363 15875 raft_consensus.cc:2454] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:32.492470 16266 raft_consensus.cc:2454] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:32.492745 15940 leader_election.cc:302] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:32.493001 17544 raft_consensus.cc:2781] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:32.493099 17544 raft_consensus.cc:482] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:32.493161 17544 raft_consensus.cc:3037] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:32.494990 17544 raft_consensus.cc:504] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:32.495507 17544 leader_election.cc:288] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Requested vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:32.495855 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0746bed2cf43b9bc7765c270a8aa62" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:32.495942 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0746bed2cf43b9bc7765c270a8aa62" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:32.496007 15875 raft_consensus.cc:3037] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:32.496136 16266 raft_consensus.cc:3037] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:32.497740 15875 raft_consensus.cc:2454] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:32.497905 16266 raft_consensus.cc:2454] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:32.498208 15940 leader_election.cc:302] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:32.498492 17544 raft_consensus.cc:2781] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:32.498641 17544 raft_consensus.cc:686] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Becoming Leader. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Running, Role: LEADER
I20240501 20:10:32.498878 17544 consensus_queue.cc:237] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:32.501914 15728 catalog_manager.cc:5484] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754fcd3b82f041568f546c6205f44272 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:32.613296 17544 raft_consensus.cc:482] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:32.613466 17544 raft_consensus.cc:504] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:32.614043 17544 leader_election.cc:288] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:32.614430 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d59d1654ef4ab3ac996f55a73b313f" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:32.614531 16135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d59d1654ef4ab3ac996f55a73b313f" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:32.614646 15875 raft_consensus.cc:2454] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:32.614756 16135 raft_consensus.cc:2454] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 0.
I20240501 20:10:32.615031 15940 leader_election.cc:302] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:32.615314 17544 raft_consensus.cc:2781] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:32.615391 17544 raft_consensus.cc:482] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:32.615470 17544 raft_consensus.cc:3037] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:32.617228 17544 raft_consensus.cc:504] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:32.617715 17544 leader_election.cc:288] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Requested vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:32.618098 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d59d1654ef4ab3ac996f55a73b313f" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:32.618182 16135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d59d1654ef4ab3ac996f55a73b313f" candidate_uuid: "754fcd3b82f041568f546c6205f44272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:32.618335 15875 raft_consensus.cc:3037] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:32.618445 16135 raft_consensus.cc:3037] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:32.621063 15875 raft_consensus.cc:2454] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:32.621068 16135 raft_consensus.cc:2454] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fcd3b82f041568f546c6205f44272 in term 1.
I20240501 20:10:32.621428 15940 leader_election.cc:302] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:32.621695 17544 raft_consensus.cc:2781] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:32.621861 17544 raft_consensus.cc:686] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Becoming Leader. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Running, Role: LEADER
I20240501 20:10:32.622115 17544 consensus_queue.cc:237] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } }
I20240501 20:10:32.625145 15728 catalog_manager.cc:5484] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754fcd3b82f041568f546c6205f44272 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:32.719667 16970 raft_consensus.cc:482] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:32.719812 16970 raft_consensus.cc:504] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:32.720350 16970 leader_election.cc:288] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:32.720680 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36465c82164a4765a63590cfd5d413a4" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:32.720732 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36465c82164a4765a63590cfd5d413a4" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:32.720880 16266 raft_consensus.cc:2454] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:32.720932 15875 raft_consensus.cc:2454] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:32.721238 16069 leader_election.cc:302] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:32.721491 16970 raft_consensus.cc:2781] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:32.721626 16970 raft_consensus.cc:482] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:32.721712 16970 raft_consensus.cc:3037] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:32.723667 16970 raft_consensus.cc:504] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:32.724186 16970 leader_election.cc:288] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Requested vote from peers f856e0d1074b431eb19763847958b89e (127.8.68.196:36919), 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:32.724479 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36465c82164a4765a63590cfd5d413a4" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:32.724625 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36465c82164a4765a63590cfd5d413a4" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:32.724700 16266 raft_consensus.cc:3037] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:32.724784 15875 raft_consensus.cc:3037] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:32.726440 16266 raft_consensus.cc:2454] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:32.726634 15875 raft_consensus.cc:2454] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:32.726799 16069 leader_election.cc:302] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:32.727041 16970 raft_consensus.cc:2781] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:32.727345 16970 raft_consensus.cc:686] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Becoming Leader. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Running, Role: LEADER
I20240501 20:10:32.727618 16970 consensus_queue.cc:237] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } }
I20240501 20:10:32.730139 15728 catalog_manager.cc:5484] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:33.087211 17567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:33.087471 17567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:33.094727 17567 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:33.098193 17544 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:33.102030 17544 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:33.108110 17544 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:33.120638 17188 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240501 20:10:33.186465 16970 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:33.194406 17565 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 216.772 ms
  time per row: 0.108386 ms
I20240501 20:10:33.495757 15728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52502:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:10:33.504406 15986 tablet_service.cc:1467] Processing CreateTablet for tablet 3643d6995910404fb8a7e6ef9c635f42 (DEFAULT_TABLE table=rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:33.504706 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 3643d6995910404fb8a7e6ef9c635f42 (DEFAULT_TABLE table=rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:33.504884 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3643d6995910404fb8a7e6ef9c635f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:33.505102 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3643d6995910404fb8a7e6ef9c635f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:33.505721 15854 tablet_service.cc:1467] Processing CreateTablet for tablet 84d7bbca7af34a2db637b55635fa2f7a (DEFAULT_TABLE table=rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:33.506086 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d7bbca7af34a2db637b55635fa2f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:33.509116 16246 tablet_service.cc:1467] Processing CreateTablet for tablet 3643d6995910404fb8a7e6ef9c635f42 (DEFAULT_TABLE table=rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:33.509483 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3643d6995910404fb8a7e6ef9c635f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:33.509792 16114 tablet_service.cc:1467] Processing CreateTablet for tablet 84d7bbca7af34a2db637b55635fa2f7a (DEFAULT_TABLE table=rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:33.510147 16114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d7bbca7af34a2db637b55635fa2f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:33.511193 16245 tablet_service.cc:1467] Processing CreateTablet for tablet eef58fcca15041a39e685a7cdcb04f2b (DEFAULT_TABLE table=rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:33.511564 16245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef58fcca15041a39e685a7cdcb04f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:33.514608 16115 tablet_service.cc:1467] Processing CreateTablet for tablet eef58fcca15041a39e685a7cdcb04f2b (DEFAULT_TABLE table=rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:33.514974 16115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef58fcca15041a39e685a7cdcb04f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:33.518782 17628 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:33.518932 15984 tablet_service.cc:1467] Processing CreateTablet for tablet eef58fcca15041a39e685a7cdcb04f2b (DEFAULT_TABLE table=rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:33.519325 15984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef58fcca15041a39e685a7cdcb04f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:33.520635 17628 tablet_bootstrap.cc:654] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:33.521663 15985 tablet_service.cc:1467] Processing CreateTablet for tablet 84d7bbca7af34a2db637b55635fa2f7a (DEFAULT_TABLE table=rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:33.522039 15985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d7bbca7af34a2db637b55635fa2f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:33.522959 17627 tablet_bootstrap.cc:492] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2: Bootstrap starting.
I20240501 20:10:33.524191 17628 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:33.524295 17628 ts_tablet_manager.cc:1367] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 20:10:33.524564 17627 tablet_bootstrap.cc:654] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:33.524914 17628 raft_consensus.cc:348] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.525096 17628 raft_consensus.cc:374] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:33.525185 17628 raft_consensus.cc:729] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.525377 17628 consensus_queue.cc:260] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.526042 17628 ts_tablet_manager.cc:1398] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:33.530995 17627 tablet_bootstrap.cc:492] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2: No bootstrap required, opened a new log
I20240501 20:10:33.531138 17627 ts_tablet_manager.cc:1367] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240501 20:10:33.531841 17627 raft_consensus.cc:348] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:33.532034 17627 raft_consensus.cc:374] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:33.532120 17627 raft_consensus.cc:729] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b81a5ae56b4dd99af1b0167c1da1e2, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.532361 17627 consensus_queue.cc:260] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:33.534734 17627 ts_tablet_manager.cc:1398] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:10:33.535864 17631 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:33.537907 17631 tablet_bootstrap.cc:654] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:33.539880 17628 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: Bootstrap starting.
I20240501 20:10:33.541342 17628 tablet_bootstrap.cc:654] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:33.543658 17628 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: No bootstrap required, opened a new log
I20240501 20:10:33.543802 17628 ts_tablet_manager.cc:1367] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:10:33.544109 17631 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:33.544240 17631 ts_tablet_manager.cc:1367] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.008s	user 0.001s	sys 0.006s
I20240501 20:10:33.544363 17628 raft_consensus.cc:348] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.544517 17628 raft_consensus.cc:374] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:33.544584 17628 raft_consensus.cc:729] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f856e0d1074b431eb19763847958b89e, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.544781 17628 consensus_queue.cc:260] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.544878 17631 raft_consensus.cc:348] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.545054 17631 raft_consensus.cc:374] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:33.545148 17631 raft_consensus.cc:729] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.545243 17628 ts_tablet_manager.cc:1398] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240501 20:10:33.545358 17631 consensus_queue.cc:260] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.545823 17626 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:33.545972 17631 ts_tablet_manager.cc:1398] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:10:33.546247 17631 tablet_bootstrap.cc:492] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:33.547639 17631 tablet_bootstrap.cc:654] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:33.548043 17626 tablet_bootstrap.cc:654] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:33.550132 17626 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:33.550347 17626 ts_tablet_manager.cc:1367] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 20:10:33.551141 17626 raft_consensus.cc:348] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.551393 17626 raft_consensus.cc:374] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:33.551501 17626 raft_consensus.cc:729] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.551723 17626 consensus_queue.cc:260] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.552260 17631 tablet_bootstrap.cc:492] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:33.552268 17626 ts_tablet_manager.cc:1398] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:33.552394 17631 ts_tablet_manager.cc:1367] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240501 20:10:33.552645 17626 tablet_bootstrap.cc:492] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:33.552940 17631 raft_consensus.cc:348] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:33.553107 17631 raft_consensus.cc:374] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:33.553176 17631 raft_consensus.cc:729] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.553413 17631 consensus_queue.cc:260] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:33.553972 17631 ts_tablet_manager.cc:1398] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240501 20:10:33.554245 17631 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:33.554559 17626 tablet_bootstrap.cc:654] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:33.555585 17631 tablet_bootstrap.cc:654] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:33.557798 17631 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272: No bootstrap required, opened a new log
I20240501 20:10:33.557914 17631 ts_tablet_manager.cc:1367] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:10:33.558578 17631 raft_consensus.cc:348] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.558725 17631 raft_consensus.cc:374] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:33.558792 17631 raft_consensus.cc:729] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.558854 17626 tablet_bootstrap.cc:492] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:33.559002 17626 ts_tablet_manager.cc:1367] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240501 20:10:33.559002 17631 consensus_queue.cc:260] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.559468 17631 ts_tablet_manager.cc:1398] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240501 20:10:33.560159 17626 raft_consensus.cc:348] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:33.560365 17626 raft_consensus.cc:374] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:33.560449 17626 raft_consensus.cc:729] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.560678 17626 consensus_queue.cc:260] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:33.561223 17626 ts_tablet_manager.cc:1398] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:33.561494 17626 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9: Bootstrap starting.
I20240501 20:10:33.562696 17626 tablet_bootstrap.cc:654] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:33.564724 17626 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9: No bootstrap required, opened a new log
I20240501 20:10:33.564857 17626 ts_tablet_manager.cc:1367] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:10:33.565256 17477 raft_consensus.cc:482] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:33.565393 17477 raft_consensus.cc:504] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.565559 17626 raft_consensus.cc:348] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.565716 17626 raft_consensus.cc:374] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:33.565800 17626 raft_consensus.cc:729] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Initialized, Role: FOLLOWER
I20240501 20:10:33.565991 17626 consensus_queue.cc:260] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.566053 17477 leader_election.cc:288] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:33.566365 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3643d6995910404fb8a7e6ef9c635f42" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:33.566560 17626 ts_tablet_manager.cc:1398] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:33.566538 16005 raft_consensus.cc:2454] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:33.566512 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3643d6995910404fb8a7e6ef9c635f42" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9" is_pre_election: true
I20240501 20:10:33.566715 16136 raft_consensus.cc:2454] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 0.
I20240501 20:10:33.566977 16201 leader_election.cc:302] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754fcd3b82f041568f546c6205f44272, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:33.567212 17477 raft_consensus.cc:2781] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:33.567306 17477 raft_consensus.cc:482] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:33.567389 17477 raft_consensus.cc:3037] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:33.569387 17477 raft_consensus.cc:504] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.569882 17477 leader_election.cc:288] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:33.570255 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3643d6995910404fb8a7e6ef9c635f42" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:33.570425 16005 raft_consensus.cc:3037] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:33.572090 16005 raft_consensus.cc:2454] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:33.572472 16201 leader_election.cc:302] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754fcd3b82f041568f546c6205f44272, f856e0d1074b431eb19763847958b89e; no voters: 
I20240501 20:10:33.572795 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3643d6995910404fb8a7e6ef9c635f42" candidate_uuid: "f856e0d1074b431eb19763847958b89e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "042aee20cf4e4e63be3ecc013fda59c9"
I20240501 20:10:33.572981 16136 raft_consensus.cc:3037] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:33.573068 17477 raft_consensus.cc:2781] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:33.573473 17477 raft_consensus.cc:686] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Becoming Leader. State: Replica: f856e0d1074b431eb19763847958b89e, State: Running, Role: LEADER
I20240501 20:10:33.573906 17477 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.575172 16136 raft_consensus.cc:2454] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f856e0d1074b431eb19763847958b89e in term 1.
I20240501 20:10:33.576640 15728 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e reported cstate change: term changed from 0 to 1, leader changed from <none> to f856e0d1074b431eb19763847958b89e (127.8.68.196). New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:33.600751 16970 raft_consensus.cc:482] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:33.600865 16970 raft_consensus.cc:504] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.601366 16970 leader_election.cc:288] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:33.601732 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef58fcca15041a39e685a7cdcb04f2b" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:33.601791 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef58fcca15041a39e685a7cdcb04f2b" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e" is_pre_election: true
I20240501 20:10:33.601919 16005 raft_consensus.cc:2454] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:33.601960 16265 raft_consensus.cc:2454] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:33.602360 16071 leader_election.cc:302] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:33.602630 16970 raft_consensus.cc:2781] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:33.602764 16970 raft_consensus.cc:482] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:33.602856 16970 raft_consensus.cc:3037] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:33.604521 16970 raft_consensus.cc:504] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.605033 16970 leader_election.cc:288] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Requested vote from peers 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:33.605317 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef58fcca15041a39e685a7cdcb04f2b" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:33.605410 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef58fcca15041a39e685a7cdcb04f2b" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f856e0d1074b431eb19763847958b89e"
I20240501 20:10:33.605512 16005 raft_consensus.cc:3037] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:33.605537 16266 raft_consensus.cc:3037] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:33.607349 16005 raft_consensus.cc:2454] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:33.607669 16071 leader_election.cc:302] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:33.607820 16266 raft_consensus.cc:2454] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:33.607945 16970 raft_consensus.cc:2781] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:33.608179 16970 raft_consensus.cc:686] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Becoming Leader. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Running, Role: LEADER
I20240501 20:10:33.608456 16970 consensus_queue.cc:237] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } }
I20240501 20:10:33.611312 15728 catalog_manager.cc:5484] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:33.621954 16970 raft_consensus.cc:482] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:33.622141 16970 raft_consensus.cc:504] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:33.622728 16970 leader_election.cc:288] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:33.623096 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d7bbca7af34a2db637b55635fa2f7a" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" is_pre_election: true
I20240501 20:10:33.623186 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d7bbca7af34a2db637b55635fa2f7a" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272" is_pre_election: true
I20240501 20:10:33.623320 15875 raft_consensus.cc:2454] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:33.623373 16005 raft_consensus.cc:2454] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 0.
I20240501 20:10:33.623720 16071 leader_election.cc:302] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 754fcd3b82f041568f546c6205f44272; no voters: 
I20240501 20:10:33.623982 16970 raft_consensus.cc:2781] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:33.624076 16970 raft_consensus.cc:482] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:33.624164 16970 raft_consensus.cc:3037] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:33.625917 16970 raft_consensus.cc:504] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:33.626499 16970 leader_election.cc:288] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Requested vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:33.626736 15875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d7bbca7af34a2db637b55635fa2f7a" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2"
I20240501 20:10:33.626928 15875 raft_consensus.cc:3037] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:33.626933 16005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d7bbca7af34a2db637b55635fa2f7a" candidate_uuid: "042aee20cf4e4e63be3ecc013fda59c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754fcd3b82f041568f546c6205f44272"
I20240501 20:10:33.627085 16005 raft_consensus.cc:3037] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:33.629555 16005 raft_consensus.cc:2454] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:33.629546 15875 raft_consensus.cc:2454] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042aee20cf4e4e63be3ecc013fda59c9 in term 1.
I20240501 20:10:33.629909 16070 leader_election.cc:302] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2; no voters: 
I20240501 20:10:33.630144 16970 raft_consensus.cc:2781] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:33.630316 16970 raft_consensus.cc:686] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Becoming Leader. State: Replica: 042aee20cf4e4e63be3ecc013fda59c9, State: Running, Role: LEADER
I20240501 20:10:33.630558 16970 consensus_queue.cc:237] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }
I20240501 20:10:33.633052 15728 catalog_manager.cc:5484] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:33.988873 17644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:33.989137 17644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:33.996493 17644 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:34.011787 17477 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:34.029489 16970 consensus_queue.cc:1035] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:34.033195 17477 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240501 20:10:34.057173 17609 consensus_queue.cc:1035] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:34.122242 17665 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:34.129293 17609 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 210.566 ms
  time per row: 0.105283 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:10:35.260815  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:37623
--local_ip_for_outbound_sockets=127.8.68.197
--tserver_master_addrs=127.8.68.254:46529
--webserver_port=43167
--webserver_interface=127.8.68.197
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:10:35.591531 17708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:35.591771 17708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:35.591869 17708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:35.599401 17708 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:35.599509 17708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:35.599639 17708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:10:35.608243 17708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:37623
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=43167
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:35.609398 17708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:35.611292 17708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:35.622989 17717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:35.623481 17720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:35.624020 17718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:35.624558 17708 server_base.cc:1034] running on GCE node
I20240501 20:10:35.625032 17708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:35.625833 17708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:35.627008 17708 hybrid_clock.cc:648] HybridClock initialized: now 1714594235626956 us; error 45 us; skew 500 ppm
I20240501 20:10:35.627278 17708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:35.629237 17708 webserver.cc:460] Webserver started at http://127.8.68.197:43167/ using document root <none> and password file <none>
I20240501 20:10:35.629877 17708 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:35.629981 17708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:35.635030 17708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:10:35.637498 17726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:35.638753 17708 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 20:10:35.638887 17708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9a0f0037700246bdb65e05aef95fb2be"
format_stamp: "Formatted at 2024-05-01 20:10:16 on dist-test-slave-08p9"
I20240501 20:10:35.639648 17708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:35.658247 17708 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:35.658912 17708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:35.659163 17708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:35.659996 17708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:35.661294 17708 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:35.661367 17708 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:35.661482 17708 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:35.661543 17708 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:35.689801 17708 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:37623
I20240501 20:10:35.689877 17840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:37623 every 8 connection(s)
I20240501 20:10:35.691174 17708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:10:35.700435  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 17708
I20240501 20:10:35.701216  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:44869
--local_ip_for_outbound_sockets=127.8.68.198
--tserver_master_addrs=127.8.68.254:46529
--webserver_port=39375
--webserver_interface=127.8.68.198
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:35.703908 17841 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:35.704200 17841 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:35.705034 17841 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:35.706517 15731 ts_manager.cc:194] Re-registered known tserver with Master: 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623)
I20240501 20:10:35.707362 15731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:50449
W20240501 20:10:36.036737 17845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:36.036968 17845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:36.037067 17845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:36.044616 17845 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:36.044759 17845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:36.044917 17845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240501 20:10:36.053310 17845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:44869
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=39375
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:36.054365 17845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:36.055927 17845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:36.067137 17850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:36.067622 17853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:36.067987 17851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:36.068647 17845 server_base.cc:1034] running on GCE node
I20240501 20:10:36.069125 17845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:36.069934 17845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:36.071112 17845 hybrid_clock.cc:648] HybridClock initialized: now 1714594236071065 us; error 41 us; skew 500 ppm
I20240501 20:10:36.071394 17845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:36.073590 17845 webserver.cc:460] Webserver started at http://127.8.68.198:39375/ using document root <none> and password file <none>
I20240501 20:10:36.074443 17845 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:36.074584 17845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:36.079365 17845 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 20:10:36.081910 17859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:36.083060 17845 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:10:36.083201 17845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d3917663631a4479b15355381486591c"
format_stamp: "Formatted at 2024-05-01 20:10:16 on dist-test-slave-08p9"
I20240501 20:10:36.083909 17845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:36.115509 17845 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:36.116180 17845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:36.116417 17845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:36.117322 17845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:36.118867 17845 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:36.118954 17845 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:36.119081 17845 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:36.119134 17845 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:36.157086 17845 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:44869
I20240501 20:10:36.157131 17973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:44869 every 8 connection(s)
I20240501 20:10:36.158361 17845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:10:36.162209  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 17845
I20240501 20:10:36.162818  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.199:44275
--local_ip_for_outbound_sockets=127.8.68.199
--tserver_master_addrs=127.8.68.254:46529
--webserver_port=34917
--webserver_interface=127.8.68.199
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:36.171958 17974 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:36.172214 17974 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:36.172994 17974 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:36.175470 15731 ts_manager.cc:194] Re-registered known tserver with Master: d3917663631a4479b15355381486591c (127.8.68.198:44869)
I20240501 20:10:36.176394 15731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:56423
W20240501 20:10:36.491945 17977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:36.492229 17977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:36.492343 17977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:36.500042 17977 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:36.500156 17977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:36.500290 17977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.199
I20240501 20:10:36.509037 17977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.199:44275
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.68.199
--webserver_port=34917
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.199
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:36.510071 17977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:36.511766 17977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:36.523895 17982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:36.524390 17985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:36.524726 17983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:36.525259 17977 server_base.cc:1034] running on GCE node
I20240501 20:10:36.525768 17977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:36.526662 17977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:36.527871 17977 hybrid_clock.cc:648] HybridClock initialized: now 1714594236527821 us; error 45 us; skew 500 ppm
I20240501 20:10:36.528175 17977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:36.530195 17977 webserver.cc:460] Webserver started at http://127.8.68.199:34917/ using document root <none> and password file <none>
I20240501 20:10:36.530939 17977 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:36.531078 17977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:36.536445 17977 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 20:10:36.539002 17991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:36.540354 17977 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:10:36.540519 17977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "cad00548f33849e7a19cecd760e1e429"
format_stamp: "Formatted at 2024-05-01 20:10:17 on dist-test-slave-08p9"
I20240501 20:10:36.541304 17977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:36.556640 17977 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:36.557264 17977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:36.557487 17977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:36.558228 17977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:36.559545 17977 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:36.559614 17977 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:36.559696 17977 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:36.559737 17977 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:36.589408 17977 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.199:44275
I20240501 20:10:36.589437 18104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.199:44275 every 8 connection(s)
I20240501 20:10:36.590790 17977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:10:36.593966  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 17977
I20240501 20:10:36.594672  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.200:38791
--local_ip_for_outbound_sockets=127.8.68.200
--tserver_master_addrs=127.8.68.254:46529
--webserver_port=37579
--webserver_interface=127.8.68.200
--builtin_ntp_servers=127.8.68.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:36.603093 18105 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:36.603323 18105 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:36.604007 18105 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:36.605244 15731 ts_manager.cc:194] Re-registered known tserver with Master: cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275)
I20240501 20:10:36.606236 15731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.199:58647
I20240501 20:10:36.709743 17841 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
W20240501 20:10:36.917819 18108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:36.918044 18108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:36.918164 18108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:36.926319 18108 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:36.926421 18108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:36.926527 18108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.200
I20240501 20:10:36.935034 18108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.200:38791
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.68.200
--webserver_port=37579
--tserver_master_addrs=127.8.68.254:46529
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.200
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:36.936053 18108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:36.937526 18108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:36.948279 18113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:36.948860 18116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:36.949281 18114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:36.949844 18108 server_base.cc:1034] running on GCE node
I20240501 20:10:36.950392 18108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:36.951350 18108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:36.952574 18108 hybrid_clock.cc:648] HybridClock initialized: now 1714594236952534 us; error 28 us; skew 500 ppm
I20240501 20:10:36.952843 18108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:36.954865 18108 webserver.cc:460] Webserver started at http://127.8.68.200:37579/ using document root <none> and password file <none>
I20240501 20:10:36.955503 18108 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:36.955616 18108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:36.960561 18108 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240501 20:10:36.963050 18122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:36.964200 18108 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:10:36.964332 18108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "047275bda1074ac5a63325e7e1109c61"
format_stamp: "Formatted at 2024-05-01 20:10:17 on dist-test-slave-08p9"
I20240501 20:10:36.965106 18108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:36.983297 18108 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:36.983893 18108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:36.984122 18108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:36.984850 18108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:36.986353 18108 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:36.986423 18108 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:36.986538 18108 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:36.986599 18108 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:37.019199 18108 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.200:38791
I20240501 20:10:37.019243 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.200:38791 every 8 connection(s)
I20240501 20:10:37.020556 18108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:10:37.025746  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 18108
I20240501 20:10:37.033025 18235 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46529
I20240501 20:10:37.033306 18235 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:37.034029 18235 heartbeater.cc:511] Master 127.8.68.254:46529 requested a full tablet report, sending...
I20240501 20:10:37.035430 15731 ts_manager.cc:194] Re-registered known tserver with Master: 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791)
I20240501 20:10:37.036361 15731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.200:34957
I20240501 20:10:37.178665 17974 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
I20240501 20:10:37.608513 18105 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
I20240501 20:10:38.038777 18235 heartbeater.cc:503] Master 127.8.68.254:46529 was elected leader, sending a full tablet report...
I20240501 20:10:38.126945 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.155529 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:38.181280 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:38.222543 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:38.231714 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:38.277082 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.300217 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.324204 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.327044 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:38.351397 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.352331 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.366210 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:38.374384 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:38.385443 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.410248 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:38.411001 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:38.474339 16071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:46677, user_credentials={real_user=slave}} blocked reactor thread for 112804us
W20240501 20:10:38.475040 15940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:35513, user_credentials={real_user=slave}} blocked reactor thread for 51336.4us
I20240501 20:10:38.685708 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.696679 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:38.711362 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:38.733305 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:10:38.788604 16200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:35513, user_credentials={real_user=slave}} blocked reactor thread for 89369.8us
I20240501 20:10:38.803946 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.805558 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.815701 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:38.841780 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:38.971257 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.016228 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.035530 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:39.061821 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.106714 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.107896 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.143158 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.158562 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.287103 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.295818 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:39.320475 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.343735 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.367933 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.402474 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:39.406900 15939 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:36919, user_credentials={real_user=slave}} blocked reactor thread for 45817.1us
I20240501 20:10:39.445739 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.464727 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.527913 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.535873 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.564483 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:39.610792 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.616261 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.617529 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:39.638834 15810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.195:46553, user_credentials={real_user=slave}} blocked reactor thread for 87619.1us
I20240501 20:10:39.663362 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.696125 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.776310 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:39.799221 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:39.816301 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.830668 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:10:39.849956 16071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:46677, user_credentials={real_user=slave}} blocked reactor thread for 109735us
I20240501 20:10:39.865869 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.889328 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.917449 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:39.947543 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.229238 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.264977 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.290967 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.291098 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:40.354396 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.360267 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.390926 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:40.423210 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.457666 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.475106 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.478483 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.486675 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.524969 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.579962 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.623227 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.624612 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.781342 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.789799 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.790475 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.802464 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.813071 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.837143 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.841090 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.846470 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.861408 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.869899 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.881953 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:40.919059 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.922245 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:40.928624 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:40.963263 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:40.983695 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:41.318797 15874 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:41.338027 16266 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.338701 18714 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:41.344789 16136 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.346870 18714 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:41.347949 15811 consensus_peers.cc:485] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e6b15e4aebf4b1f9a3176f87083703e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.362387 18348 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:41.364063 16265 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:41.373817 15731 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:41.381248 16136 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:41.384464 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:41.392655 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:41.398209 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:41.402804 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:41.427207 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:41.462510 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:41.476343 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:41.489943 16135 consensus_queue.cc:237] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:41.498307 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:41.535116 16005 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.535327 16266 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.535811 18815 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:41.536489 18815 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:41.564050 18815 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:41.568397 16265 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:41.568732 16005 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
W20240501 20:10:41.574012 16072 consensus_peers.cc:485] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5682259bc54c469e9c6a08723c1bbee0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.574723 15726 catalog_manager.cc:5484] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:41.648329 16005 consensus_queue.cc:237] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:41.651779 16266 raft_consensus.cc:1261] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.652366 18549 consensus_queue.cc:1035] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:41.686170 15874 raft_consensus.cc:1261] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.687026 18549 consensus_queue.cc:1035] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:41.699568 18549 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:41.700150 16266 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
W20240501 20:10:41.704110 15940 consensus_peers.cc:485] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f25f3a86f0b494b89fe8f86aae4f4cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.710352 15727 catalog_manager.cc:5484] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:41.711040 15874 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:41.748936 18911 ts_tablet_manager.cc:902] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:41.770655 15875 consensus_queue.cc:237] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:41.784039 16135 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.784976 18852 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:41.788096 16005 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.789180 18845 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:41.796950 15808 consensus_peers.cc:485] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fce37f2e8f74a119b8bd0c34b1a35ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.829968 18348 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:41.830943 16005 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:41.838328 15727 catalog_manager.cc:5484] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:41.855230 16136 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:41.893195 18911 tablet_copy_client.cc:308] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:41.922302 15875 consensus_queue.cc:237] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:41.925917 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet 2e6b15e4aebf4b1f9a3176f87083703e from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:37603)
I20240501 20:10:41.926198 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet 2e6b15e4aebf4b1f9a3176f87083703e from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:37603: session id = cad00548f33849e7a19cecd760e1e429-2e6b15e4aebf4b1f9a3176f87083703e
I20240501 20:10:41.930889 15895 tablet_copy_source_session.cc:205] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:41.937314 16266 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.938025 18845 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:41.951485 15811 consensus_peers.cc:485] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 627e9e5c120747f1916855ea83008657. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.957018 18911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6b15e4aebf4b1f9a3176f87083703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:41.959611 16005 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:41.960278 18845 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:41.992210 18845 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:41.994863 16004 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:42.000437 15728 catalog_manager.cc:5484] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:42.004482 16265 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:42.011200 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:42.069541 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:42.078214 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.083652 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.109620 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:42.110070 19005 ts_tablet_manager.cc:902] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:42.129710 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:42.161288 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.162274 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.167829 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:42.194619 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.204450 19008 ts_tablet_manager.cc:902] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:42.214807 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.230373 18911 tablet_copy_client.cc:783] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:42.230831 18911 tablet_copy_client.cc:647] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:42.233326 18911 tablet_copy_client.cc:523] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:42.240433 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:42.247551 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:42.253765 18911 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:42.254211 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.262352 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.271749 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:42.272958 16266 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:42.285490 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:42.339273 16005 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:42.339884 18877 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:42.352356 16136 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:42.352988 18877 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:42.408185 19033 ts_tablet_manager.cc:902] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:42.408843 19008 tablet_copy_client.cc:308] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:42.418776 19039 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:42.429968 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.429912 16005 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:42.430759 19005 tablet_copy_client.cc:308] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:42.433652 16135 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:42.439260 15727 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:42.461961 18911 log.cc:824] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:42.462836 19049 ts_tablet_manager.cc:902] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:42.472235 19033 tablet_copy_client.cc:308] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:42.485110 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.490012 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:42.492547 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.505143 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet 5682259bc54c469e9c6a08723c1bbee0 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:53423)
I20240501 20:10:42.505411 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet 5682259bc54c469e9c6a08723c1bbee0 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:53423: session id = 9a0f0037700246bdb65e05aef95fb2be-5682259bc54c469e9c6a08723c1bbee0
I20240501 20:10:42.509080 16156 tablet_copy_source_session.cc:205] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:42.511770 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:42.514962 19008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5682259bc54c469e9c6a08723c1bbee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:42.524430 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:10:42.535959 16202 consensus_peers.cc:485] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3643d6995910404fb8a7e6ef9c635f42. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:42.538102 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:42.550457 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:49021)
I20240501 20:10:42.550743 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet 0f25f3a86f0b494b89fe8f86aae4f4cd from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:49021: session id = 047275bda1074ac5a63325e7e1109c61-0f25f3a86f0b494b89fe8f86aae4f4cd
I20240501 20:10:42.554687 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:60703)
I20240501 20:10:42.554868 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet 2fce37f2e8f74a119b8bd0c34b1a35ac from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:60703: session id = d3917663631a4479b15355381486591c-2fce37f2e8f74a119b8bd0c34b1a35ac
I20240501 20:10:42.556758 16026 tablet_copy_source_session.cc:205] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:42.557509 15895 tablet_copy_source_session.cc:205] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:42.561928 19005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:42.562028 19033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:42.563629 19008 tablet_copy_client.cc:783] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:42.563942 19008 tablet_copy_client.cc:647] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:42.576874 19008 tablet_copy_client.cc:523] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:42.593272 19008 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:42.612792 19049 tablet_copy_client.cc:308] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:42.620003 19005 tablet_copy_client.cc:783] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:42.620342 19005 tablet_copy_client.cc:647] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:42.629772 16005 consensus_queue.cc:237] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:42.635360 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet 627e9e5c120747f1916855ea83008657 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:49547)
I20240501 20:10:42.635531 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet 627e9e5c120747f1916855ea83008657 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:49547: session id = 9a0f0037700246bdb65e05aef95fb2be-627e9e5c120747f1916855ea83008657
I20240501 20:10:42.636785 19008 log.cc:824] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:42.639138 15895 tablet_copy_source_session.cc:205] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:42.640523 19049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627e9e5c120747f1916855ea83008657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:42.642060 19005 tablet_copy_client.cc:523] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:42.650184 19033 tablet_copy_client.cc:783] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:42.651885 19033 tablet_copy_client.cc:647] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:42.657680 19049 tablet_copy_client.cc:783] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:42.657980 19049 tablet_copy_client.cc:647] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:42.658085 19033 tablet_copy_client.cc:523] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:42.661763 19049 tablet_copy_client.cc:523] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:42.677397 16136 raft_consensus.cc:1261] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:42.678087 18909 consensus_queue.cc:1035] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:42.679530 19033 tablet_bootstrap.cc:492] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:42.688320 15875 raft_consensus.cc:1261] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:42.688995 19059 consensus_queue.cc:1035] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:42.714565 18909 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:42.719280 16135 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:42.725960 15728 catalog_manager.cc:5484] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:42.734082 19033 log.cc:824] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:42.734344 19049 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:42.745292 15875 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:42.757440 19005 tablet_bootstrap.cc:492] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
W20240501 20:10:42.764709 15939 consensus_peers.cc:485] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7fb9bc4cd73492da44bde1eecd627dc. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:42.790403 19033 tablet_bootstrap.cc:492] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:42.790901 19033 tablet_bootstrap.cc:492] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:42.791357 19033 ts_tablet_manager.cc:1367] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.112s	user 0.058s	sys 0.004s
I20240501 20:10:42.797333 19033 raft_consensus.cc:348] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:42.797792 19033 raft_consensus.cc:729] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:42.798331 19033 consensus_queue.cc:260] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:42.806509 19033 ts_tablet_manager.cc:1398] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.015s	user 0.014s	sys 0.001s
I20240501 20:10:42.807382 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session d3917663631a4479b15355381486591c-2fce37f2e8f74a119b8bd0c34b1a35ac received from {username='slave'} at 127.8.68.198:60703
I20240501 20:10:42.807551 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session d3917663631a4479b15355381486591c-2fce37f2e8f74a119b8bd0c34b1a35ac on tablet 2fce37f2e8f74a119b8bd0c34b1a35ac with peer d3917663631a4479b15355381486591c
I20240501 20:10:42.824818 19008 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:42.826872 19008 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:42.827330 19008 ts_tablet_manager.cc:1367] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.234s	user 0.054s	sys 0.013s
I20240501 20:10:42.833732 19008 raft_consensus.cc:348] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:42.848356 18911 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:42.848912 18911 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:42.849342 18911 ts_tablet_manager.cc:1367] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.596s	user 0.059s	sys 0.002s
I20240501 20:10:42.834231 19008 raft_consensus.cc:729] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:42.852303 19008 consensus_queue.cc:260] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:42.853190 19049 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:42.853637 19049 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:42.853916 19049 ts_tablet_manager.cc:1367] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.152s	user 0.049s	sys 0.000s
I20240501 20:10:42.854924 19049 raft_consensus.cc:348] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:42.855197 19049 raft_consensus.cc:729] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:42.855412 19049 consensus_queue.cc:260] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:42.868439 19005 log.cc:824] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:42.874241 19049 ts_tablet_manager.cc:1398] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.020s	user 0.011s	sys 0.008s
I20240501 20:10:42.874828 18911 raft_consensus.cc:348] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:42.875171 18911 raft_consensus.cc:729] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:42.875681 18911 consensus_queue.cc:260] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:42.876273 16005 consensus_queue.cc:237] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:42.878666 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-627e9e5c120747f1916855ea83008657 received from {username='slave'} at 127.8.68.197:49547
I20240501 20:10:42.878813 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-627e9e5c120747f1916855ea83008657 on tablet 627e9e5c120747f1916855ea83008657 with peer 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.879129 15940 consensus_peers.cc:485] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f904b62401ff4528b3fa65dd4e11512a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:42.880719 16266 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:42.881613 19098 ts_tablet_manager.cc:902] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:42.882189 18908 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:42.883867 16136 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:42.884841 18909 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:42.886705 18911 ts_tablet_manager.cc:1398] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.037s	user 0.012s	sys 0.000s
I20240501 20:10:42.887199 19008 ts_tablet_manager.cc:1398] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.060s	user 0.008s	sys 0.011s
I20240501 20:10:42.889256 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-2e6b15e4aebf4b1f9a3176f87083703e received from {username='slave'} at 127.8.68.199:37603
I20240501 20:10:42.889353 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-5682259bc54c469e9c6a08723c1bbee0 received from {username='slave'} at 127.8.68.197:53423
I20240501 20:10:42.889410 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session cad00548f33849e7a19cecd760e1e429-2e6b15e4aebf4b1f9a3176f87083703e on tablet 2e6b15e4aebf4b1f9a3176f87083703e with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:42.889511 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-5682259bc54c469e9c6a08723c1bbee0 on tablet 5682259bc54c469e9c6a08723c1bbee0 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:42.898330 18909 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:42.903227 16265 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:42.910756 19098 tablet_copy_client.cc:308] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:42.914398 15727 catalog_manager.cc:5484] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:42.917589 16136 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:42.970964 16286 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet 3643d6995910404fb8a7e6ef9c635f42 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:39585)
I20240501 20:10:42.971201 16286 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet 3643d6995910404fb8a7e6ef9c635f42 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:39585: session id = cad00548f33849e7a19cecd760e1e429-3643d6995910404fb8a7e6ef9c635f42
I20240501 20:10:42.976277 16286 tablet_copy_source_session.cc:205] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:42.978071 19098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3643d6995910404fb8a7e6ef9c635f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:42.986320 19098 tablet_copy_client.cc:783] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:42.986972 19098 tablet_copy_client.cc:647] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:42.989102 19098 tablet_copy_client.cc:523] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:42.989722 16136 consensus_queue.cc:237] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:42.993582 19098 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:42.994475 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:43.011152 18060 raft_consensus.cc:1203] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:43.013689 17929 raft_consensus.cc:1203] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:43.015690 16266 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:43.016618 16072 consensus_peers.cc:485] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36465c82164a4765a63590cfd5d413a4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:43.016912 19070 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.017073 15875 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.017695 19070 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.040141 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:43.050422 19070 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:43.055168 16265 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:43.058205 17795 raft_consensus.cc:1203] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:43.063599 15727 catalog_manager.cc:5484] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:43.080915 15875 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:43.103935 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:43.107424 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:43.139387 19033 ts_tablet_manager.cc:902] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:43.163383 19033 tablet_copy_client.cc:308] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:43.171969 19098 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:43.172418 19098 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:43.172722 19098 ts_tablet_manager.cc:1367] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.179s	user 0.045s	sys 0.003s
I20240501 20:10:43.173754 19098 raft_consensus.cc:348] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:43.174016 19098 raft_consensus.cc:729] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:43.186812 17795 raft_consensus.cc:1203] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:43.174222 19098 consensus_queue.cc:260] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:43.193015 19098 ts_tablet_manager.cc:1398] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.000s
I20240501 20:10:43.193934 16286 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-3643d6995910404fb8a7e6ef9c635f42 received from {username='slave'} at 127.8.68.199:39585
I20240501 20:10:43.194118 16286 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session cad00548f33849e7a19cecd760e1e429-3643d6995910404fb8a7e6ef9c635f42 on tablet 3643d6995910404fb8a7e6ef9c635f42 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:43.203701 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet f7fb9bc4cd73492da44bde1eecd627dc from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:50675)
I20240501 20:10:43.203891 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet f7fb9bc4cd73492da44bde1eecd627dc from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:50675: session id = d3917663631a4479b15355381486591c-f7fb9bc4cd73492da44bde1eecd627dc
I20240501 20:10:43.210448 16026 tablet_copy_source_session.cc:205] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:43.210572 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:43.212357 15874 consensus_queue.cc:237] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:43.222851 19033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7fb9bc4cd73492da44bde1eecd627dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:43.223821 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:43.241827 19033 tablet_copy_client.cc:783] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:43.242111 19033 tablet_copy_client.cc:647] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:43.255126 17929 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.255923 18348 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.263857 16005 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.264434 18348 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.266700 18348 raft_consensus.cc:1013] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: attempt to promote peer d3917663631a4479b15355381486591c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:10:43.267678 18852 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.272161 16005 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.281821 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:43.282387 15727 catalog_manager.cc:5484] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.282050 17929 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.287407 18348 raft_consensus.cc:1050] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:43.288092 16136 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.288416 18348 consensus_queue.cc:237] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:43.291157 17928 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:43.291754 18845 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:43.295171 16136 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:43.295774 16005 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:43.296715 18852 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.297111 18348 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:43.298719 16136 raft_consensus.cc:858] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }: 
  New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
W20240501 20:10:43.298822 16136 raft_consensus.cc:1565] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }: 
  New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
W20240501 20:10:43.308964 15809 consensus_peers.cc:485] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fce37f2e8f74a119b8bd0c34b1a35ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:43.312999 19033 tablet_copy_client.cc:523] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:43.316236 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:43.316525 19033 tablet_bootstrap.cc:492] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:43.322743 19162 ts_tablet_manager.cc:902] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:43.325379 16266 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:43.325575 18348 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.328267 18060 raft_consensus.cc:1203] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:43.328476 16136 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.330859 17929 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.335312 16005 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.335597 15726 catalog_manager.cc:5484] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 5 to 6, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:43.356719 19005 tablet_bootstrap.cc:492] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:43.357368 19005 tablet_bootstrap.cc:492] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:43.357862 19005 ts_tablet_manager.cc:1367] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.601s	user 0.066s	sys 0.000s
I20240501 20:10:43.363636 15874 consensus_queue.cc:237] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:43.364354 19005 raft_consensus.cc:348] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:43.364873 19005 raft_consensus.cc:729] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:43.365468 19005 consensus_queue.cc:260] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:43.365978 17929 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:43.366358 19162 tablet_copy_client.cc:308] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:43.367017 16005 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:43.367702 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet f904b62401ff4528b3fa65dd4e11512a from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:49021)
I20240501 20:10:43.367846 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet f904b62401ff4528b3fa65dd4e11512a from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:49021: session id = 047275bda1074ac5a63325e7e1109c61-f904b62401ff4528b3fa65dd4e11512a
I20240501 20:10:43.371052 18852 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:43.371646 18852 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:43.373878 19005 ts_tablet_manager.cc:1398] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.016s	user 0.016s	sys 0.000s
I20240501 20:10:43.375416 16025 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-0f25f3a86f0b494b89fe8f86aae4f4cd received from {username='slave'} at 127.8.68.200:49021
I20240501 20:10:43.375581 16025 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-0f25f3a86f0b494b89fe8f86aae4f4cd on tablet 0f25f3a86f0b494b89fe8f86aae4f4cd with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:43.376969 18348 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:43.379666 17929 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:43.381922 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:43.384833 16005 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:43.388758 15726 catalog_manager.cc:5484] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:43.392853 16026 tablet_copy_source_session.cc:205] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:43.394035 19162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f904b62401ff4528b3fa65dd4e11512a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:43.404546 19162 tablet_copy_client.cc:783] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:43.404954 19162 tablet_copy_client.cc:647] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:43.408488 19162 tablet_copy_client.cc:523] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:43.416498 19162 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:43.417199 15874 consensus_queue.cc:237] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:43.419095 16005 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:43.419761 18348 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:43.420506 17929 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:43.422096 18348 raft_consensus.cc:1050] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:43.422868 18845 consensus_queue.cc:1035] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:43.422967 18348 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:43.425230 18852 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
W20240501 20:10:43.426833 16202 consensus_peers.cc:485] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c538581c71f646b5bc8d71cb6981f201. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:43.428161 16004 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.428773 16134 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.429688 19159 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.430244 19159 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.432286 17929 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:43.440315 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:43.442344 15726 catalog_manager.cc:5484] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 7 to 8, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:43.455255 16003 raft_consensus.cc:2932] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:43.459374 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:43.470384 16003 consensus_queue.cc:237] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:43.473269 18170 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac with delete_type TABLET_DATA_TOMBSTONED (TS 047275bda1074ac5a63325e7e1109c61 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35670
I20240501 20:10:43.473861 19183 tablet_replica.cc:330] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:43.473935 16135 raft_consensus.cc:1261] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:43.474381 19183 raft_consensus.cc:2229] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:43.474438 16133 raft_consensus.cc:1261] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.474675 19183 raft_consensus.cc:2258] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:43.474977 18909 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.476761 19183 ts_tablet_manager.cc:1875] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:10:43.478106 15939 consensus_peers.cc:485] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67d59d1654ef4ab3ac996f55a73b313f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:43.478876 19159 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:43.485005 19183 ts_tablet_manager.cc:1888] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:43.485224 19183 log.cc:1196] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/2fce37f2e8f74a119b8bd0c34b1a35ac
I20240501 20:10:43.486292 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet 2fce37f2e8f74a119b8bd0c34b1a35ac (table rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]) successfully deleted
W20240501 20:10:43.487242 15719 catalog_manager.cc:4797] TS 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): delete failed for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac because the tablet was not found. No further retry: Not found: Tablet not found: 2fce37f2e8f74a119b8bd0c34b1a35ac
I20240501 20:10:43.487911 15874 raft_consensus.cc:1261] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.488641 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fce37f2e8f74a119b8bd0c34b1a35ac with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:43.489128 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet 2fce37f2e8f74a119b8bd0c34b1a35ac (table rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]) successfully deleted
I20240501 20:10:43.490880 18060 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.491561 18845 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.485263 15727 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:43.498911 16133 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.499482 18845 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.502836 16266 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.503471 19182 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.511698 18909 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20240501 20:10:43.513442 18845 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.519084 18060 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.525223 15727 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.525985 19033 tablet_bootstrap.cc:492] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:43.526423 19033 tablet_bootstrap.cc:492] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:43.526717 19033 ts_tablet_manager.cc:1367] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.210s	user 0.048s	sys 0.000s
I20240501 20:10:43.527659 19033 raft_consensus.cc:348] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:43.531219 19033 raft_consensus.cc:729] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:43.531431 19033 consensus_queue.cc:260] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:43.538969 19033 ts_tablet_manager.cc:1398] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240501 20:10:43.539508 16026 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session d3917663631a4479b15355381486591c-f7fb9bc4cd73492da44bde1eecd627dc received from {username='slave'} at 127.8.68.198:50675
I20240501 20:10:43.545886 16266 consensus_queue.cc:237] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:43.546875 15874 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:43.550344 18060 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:43.551059 18348 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:43.552428 16026 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session d3917663631a4479b15355381486591c-f7fb9bc4cd73492da44bde1eecd627dc on tablet f7fb9bc4cd73492da44bde1eecd627dc with peer d3917663631a4479b15355381486591c
I20240501 20:10:43.553342 18852 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.511417 16003 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:43.557224 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e6b15e4aebf4b1f9a3176f87083703e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:43.560276 15727 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:43.563274 16133 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.568126 16134 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:43.570169 18909 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:43.570616 16003 consensus_queue.cc:237] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:43.572614 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6b15e4aebf4b1f9a3176f87083703e with delete_type TABLET_DATA_TOMBSTONED (TS f856e0d1074b431eb19763847958b89e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44866
I20240501 20:10:43.569180 16133 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:43.573388 18060 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.581931 15727 catalog_manager.cc:5484] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:43.585598 16136 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:43.591096 19182 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20240501 20:10:43.591101 15875 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:43.599350 18190 raft_consensus.cc:1203] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:43.600836 17929 raft_consensus.cc:1203] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:43.611207 16136 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.619380 19208 tablet_replica.cc:330] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: stopping tablet replica
I20240501 20:10:43.619904 19208 raft_consensus.cc:2229] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:43.621042 18815 raft_consensus.cc:1050] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:43.621975 18815 consensus_queue.cc:237] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:43.626502 19208 pending_rounds.cc:62] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: Trying to abort 1 pending ops.
I20240501 20:10:43.626716 19208 pending_rounds.cc:68] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022978021058367488 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2e6b15e4aebf4b1f9a3176f87083703e" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } } }
I20240501 20:10:43.627691 19208 raft_consensus.cc:2894] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:10:43.627863 19208 raft_consensus.cc:2866] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:43.628754 19208 raft_consensus.cc:2258] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:43.630826 19208 ts_tablet_manager.cc:1875] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:10:43.631705 15939 consensus_peers.cc:485] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3761f02c20a4448acd89f4a37f1193c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:43.633244 15875 raft_consensus.cc:1261] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.633839 19059 consensus_queue.cc:1035] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.632927 16265 raft_consensus.cc:1261] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.637327 19059 consensus_queue.cc:1035] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.641369 19210 ts_tablet_manager.cc:902] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:43.641695 16003 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.642230 19031 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.644963 15874 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.647173 16265 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.648692 19031 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.649957 19059 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:43.654850 15874 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:43.657198 15725 catalog_manager.cc:5484] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:43.673717 19031 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:43.677843 15873 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:43.678265 19208 ts_tablet_manager.cc:1888] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:43.678428 19208 log.cc:1196] T 2e6b15e4aebf4b1f9a3176f87083703e P f856e0d1074b431eb19763847958b89e: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/2e6b15e4aebf4b1f9a3176f87083703e
I20240501 20:10:43.680580 15718 catalog_manager.cc:4830] TS f856e0d1074b431eb19763847958b89e (127.8.68.196:36919): tablet 2e6b15e4aebf4b1f9a3176f87083703e (table rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]) successfully deleted
I20240501 20:10:43.684608 15727 catalog_manager.cc:5484] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:43.689162 15873 consensus_queue.cc:237] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:43.698513 16003 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.699100 18815 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.701714 19162 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:43.702124 19162 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:43.707300 19162 ts_tablet_manager.cc:1367] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.291s	user 0.046s	sys 0.024s
I20240501 20:10:43.708312 19162 raft_consensus.cc:348] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:43.708570 19162 raft_consensus.cc:729] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:43.708782 19162 consensus_queue.cc:260] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:43.709311 19162 ts_tablet_manager.cc:1398] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:10:43.709630 18815 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.709704 16005 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:43.715842 16026 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-f904b62401ff4528b3fa65dd4e11512a received from {username='slave'} at 127.8.68.200:49021
I20240501 20:10:43.711418 17795 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.715973 16026 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-f904b62401ff4528b3fa65dd4e11512a on tablet f904b62401ff4528b3fa65dd4e11512a with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:43.720139 18815 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.724925 16264 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:43.728843 18852 raft_consensus.cc:1050] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:43.729616 18852 consensus_queue.cc:237] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:43.740489 19067 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
W20240501 20:10:43.744518 16202 consensus_peers.cc:485] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b016fc1d28524a02bd762b74c6b08375. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:43.744985 17795 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.745620 16266 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.746127 15727 catalog_manager.cc:5484] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:10:43.751930 15809 consensus_peers.cc:485] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9a1a818ab7e44a5ac5c67f42e1dfe3b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:43.752393 16134 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.753003 18348 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.753365 16004 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.753921 18348 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.770680 16266 consensus_queue.cc:237] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:43.780025 19210 tablet_copy_client.cc:308] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:43.780699 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet 36465c82164a4765a63590cfd5d413a4 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:53423)
I20240501 20:10:43.780870 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet 36465c82164a4765a63590cfd5d413a4 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:53423: session id = 9a0f0037700246bdb65e05aef95fb2be-36465c82164a4765a63590cfd5d413a4
I20240501 20:10:43.787120 16156 tablet_copy_source_session.cc:205] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:43.788580 16136 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.789212 19031 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:43.789660 15873 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:43.790971 19210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36465c82164a4765a63590cfd5d413a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:43.794967 19031 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:43.800480 16200 consensus_peers.cc:485] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb90224d08f84a2faf7dcc063c6af56e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:43.803951 19210 tablet_copy_client.cc:783] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:43.814556 19159 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.815213 16133 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.818432 19210 tablet_copy_client.cc:647] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:43.820788 15725 catalog_manager.cc:5484] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:43.826656 15873 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.830643 19210 tablet_copy_client.cc:523] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:43.838225 18348 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.845402 15725 catalog_manager.cc:5484] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:43.852263 16004 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.857424 17794 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.858176 19182 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.864262 16134 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:43.864666 16136 consensus_queue.cc:237] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:43.866662 16266 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:43.867247 16266 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.867242 19067 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:43.868324 16004 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.868899 18348 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.871443 18348 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.882026 18190 raft_consensus.cc:1203] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:43.888175 19210 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:43.893283 19182 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.900038 15727 catalog_manager.cc:5484] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:43.905370 16264 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.907958 19159 raft_consensus.cc:1050] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:43.908968 19159 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:43.911944 18060 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.912519 19166 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.917156 16004 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.917027 15873 consensus_queue.cc:237] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:43.917789 19159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.917726 16005 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.898025 17794 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
W20240501 20:10:43.932665 16071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:46677, user_credentials={real_user=slave}} blocked reactor thread for 45278.7us
I20240501 20:10:43.933400 16136 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:43.934312 19159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:43.951642 18815 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.952981 16264 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.958916 15727 catalog_manager.cc:5484] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.966336 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet 5682259bc54c469e9c6a08723c1bbee0 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:43.976799 19269 tablet_replica.cc:330] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:43.979991 19159 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.981532 17792 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:43.982904 18348 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:43.984483 18060 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.985700 18852 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.986183 17792 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:43.988965 16004 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.990018 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 627e9e5c120747f1916855ea83008657 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:43.991498 15727 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:43.992045 15726 catalog_manager.cc:5484] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:43.995462 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5682259bc54c469e9c6a08723c1bbee0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:43.997304 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet 627e9e5c120747f1916855ea83008657 with delete_type TABLET_DATA_TOMBSTONED (TS f856e0d1074b431eb19763847958b89e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44866
I20240501 20:10:43.997761 19208 tablet_replica.cc:330] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e: stopping tablet replica
I20240501 20:10:43.997975 19208 raft_consensus.cc:2229] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:43.998198 19208 raft_consensus.cc:2258] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:44.000602 19269 raft_consensus.cc:2229] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:44.003868 19269 pending_rounds.cc:62] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: Trying to abort 1 pending ops.
I20240501 20:10:44.004065 19269 pending_rounds.cc:68] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022978021873233920 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5682259bc54c469e9c6a08723c1bbee0" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } } }
I20240501 20:10:44.004818 19208 ts_tablet_manager.cc:1875] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:44.004987 19269 raft_consensus.cc:2894] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:10:44.005193 19269 raft_consensus.cc:2866] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:44.005371 19269 raft_consensus.cc:2258] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:44.011524 19208 ts_tablet_manager.cc:1888] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:44.011740 19208 log.cc:1196] T 627e9e5c120747f1916855ea83008657 P f856e0d1074b431eb19763847958b89e: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/627e9e5c120747f1916855ea83008657
I20240501 20:10:44.012596 15718 catalog_manager.cc:4830] TS f856e0d1074b431eb19763847958b89e (127.8.68.196:36919): tablet 627e9e5c120747f1916855ea83008657 (table rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]) successfully deleted
I20240501 20:10:44.015697 16136 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:44.023551 19033 ts_tablet_manager.cc:902] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:44.041249 19269 ts_tablet_manager.cc:1875] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:44.047482 19277 ts_tablet_manager.cc:902] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:44.048863 18909 raft_consensus.cc:1050] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:44.049974 18909 consensus_queue.cc:237] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:44.058853 16003 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.060027 18852 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:44.063603 16005 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:44.066183 17929 raft_consensus.cc:1261] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.065490 19269 ts_tablet_manager.cc:1888] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:44.066919 18908 consensus_queue.cc:1035] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.067893 19269 log.cc:1196] T 5682259bc54c469e9c6a08723c1bbee0 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/5682259bc54c469e9c6a08723c1bbee0
I20240501 20:10:44.069370 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet 5682259bc54c469e9c6a08723c1bbee0 (table rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]) successfully deleted
I20240501 20:10:44.072400 15873 raft_consensus.cc:1261] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.073200 19059 consensus_queue.cc:1035] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.075443 18908 raft_consensus.cc:1050] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
I20240501 20:10:44.076216 18908 consensus_queue.cc:237] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:44.078502 17793 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.080801 19275 ts_tablet_manager.cc:902] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:44.091902 16266 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:44.093766 18059 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.094316 19159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:44.097040 19031 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:44.102033 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3643d6995910404fb8a7e6ef9c635f42 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:44.102972 18059 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:44.103667 19070 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:44.105535 17793 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:44.109519 15728 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:44.115834 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet 3643d6995910404fb8a7e6ef9c635f42 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:44.116242 19269 tablet_replica.cc:330] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:44.116429 19269 raft_consensus.cc:2229] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:44.116655 19269 raft_consensus.cc:2258] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:44.118000 19269 ts_tablet_manager.cc:1875] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:44.126505 19269 ts_tablet_manager.cc:1888] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:44.126708 19269 log.cc:1196] T 3643d6995910404fb8a7e6ef9c635f42 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/3643d6995910404fb8a7e6ef9c635f42
I20240501 20:10:44.127543 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet 3643d6995910404fb8a7e6ef9c635f42 (table rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]) successfully deleted
I20240501 20:10:44.142136 16134 raft_consensus.cc:1261] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.142796 19283 consensus_queue.cc:1035] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.150148 16133 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.150863 19031 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:44.152315 16133 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:44.165599 18909 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:44.166841 15874 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:44.171295 15875 consensus_queue.cc:237] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:44.185192 19033 tablet_copy_client.cc:308] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:44.185878 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet 67d59d1654ef4ab3ac996f55a73b313f from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:50675)
I20240501 20:10:44.186051 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet 67d59d1654ef4ab3ac996f55a73b313f from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:50675: session id = d3917663631a4479b15355381486591c-67d59d1654ef4ab3ac996f55a73b313f
I20240501 20:10:44.188999 16026 tablet_copy_source_session.cc:205] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:44.190403 19033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d59d1654ef4ab3ac996f55a73b313f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:44.195430 16136 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:44.196112 19301 ts_tablet_manager.cc:902] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:44.199697 19275 tablet_copy_client.cc:308] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:44.199844 15874 raft_consensus.cc:1261] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.200294 16266 raft_consensus.cc:1261] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.200397 19059 consensus_queue.cc:1035] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.200448 16286 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet c538581c71f646b5bc8d71cb6981f201 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:39585)
I20240501 20:10:44.200627 16286 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet c538581c71f646b5bc8d71cb6981f201 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:39585: session id = cad00548f33849e7a19cecd760e1e429-c538581c71f646b5bc8d71cb6981f201
I20240501 20:10:44.200978 18190 raft_consensus.cc:1261] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.203536 16286 tablet_copy_source_session.cc:205] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:44.204284 19162 ts_tablet_manager.cc:902] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:44.204859 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c538581c71f646b5bc8d71cb6981f201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:44.206796 15727 catalog_manager.cc:5484] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:44.207463 19059 consensus_queue.cc:1035] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.207908 19059 consensus_queue.cc:1035] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.218387 16005 consensus_queue.cc:237] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:44.220806 17928 raft_consensus.cc:1261] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.223412 19059 consensus_queue.cc:1035] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.224649 17927 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:44.234233 16134 consensus_queue.cc:237] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:44.242413 16264 consensus_queue.cc:237] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:44.245101 16005 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:10:44.245348 16200 consensus_peers.cc:485] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b016fc1d28524a02bd762b74c6b08375. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:44.245488 16202 consensus_peers.cc:485] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b016fc1d28524a02bd762b74c6b08375. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:44.245713 19166 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.245713 18909 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.245174 15872 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.248153 19159 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.248548 19159 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:44.251274 16005 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:44.253975 15874 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.255200 15728 catalog_manager.cc:5484] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:44.258476 19033 tablet_copy_client.cc:783] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:44.258709 19033 tablet_copy_client.cc:647] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:44.264060 15873 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:44.264979 15728 catalog_manager.cc:5484] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:44.268728 19301 tablet_copy_client.cc:308] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:44.269469 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet f3761f02c20a4448acd89f4a37f1193c from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:50675)
I20240501 20:10:44.269615 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet f3761f02c20a4448acd89f4a37f1193c from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:50675: session id = d3917663631a4479b15355381486591c-f3761f02c20a4448acd89f4a37f1193c
I20240501 20:10:44.270069 16266 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.272394 16026 tablet_copy_source_session.cc:205] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:44.273317 19312 ts_tablet_manager.cc:902] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:44.275028 19301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3761f02c20a4448acd89f4a37f1193c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:44.278993 18189 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.281778 16005 consensus_queue.cc:237] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:44.282927 19275 tablet_copy_client.cc:783] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:44.283185 19275 tablet_copy_client.cc:647] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:44.287422 18908 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:44.287671 17927 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:44.289311 19277 tablet_copy_client.cc:308] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:44.291370 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f7fb9bc4cd73492da44bde1eecd627dc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:44.291644 16005 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:44.292585 19067 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:44.292807 16264 raft_consensus.cc:1261] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.296092 19283 consensus_queue.cc:1035] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240501 20:10:44.296978 19033 tablet_copy_client.cc:523] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:44.299893 19275 tablet_copy_client.cc:523] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:44.300204 19033 tablet_bootstrap.cc:492] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:44.301232 16264 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:44.301995 18815 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:44.303933 19275 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
W20240501 20:10:44.304142 15808 consensus_peers.cc:485] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c74a6da428b04df58b7edcdf88d81cca. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:44.305953 16134 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:44.307818 16263 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:44.308732 19198 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:44.309895 18348 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:10:44.314736 15727 catalog_manager.cc:5484] T f7fb9bc4cd73492da44bde1eecd627dc P d3917663631a4479b15355381486591c reported cstate change: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:44.318666 15854 tablet_service.cc:1514] Processing DeleteTablet for tablet f7fb9bc4cd73492da44bde1eecd627dc with delete_type TABLET_DATA_TOMBSTONED (TS 51b81a5ae56b4dd99af1b0167c1da1e2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35708
I20240501 20:10:44.319662 19059 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.321480 19301 tablet_copy_client.cc:783] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:44.321768 19301 tablet_copy_client.cc:647] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:44.323251 16286 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet b016fc1d28524a02bd762b74c6b08375 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:55615)
I20240501 20:10:44.323387 16286 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet b016fc1d28524a02bd762b74c6b08375 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:55615: session id = 9a0f0037700246bdb65e05aef95fb2be-b016fc1d28524a02bd762b74c6b08375
I20240501 20:10:44.323928 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:44.325296 18190 raft_consensus.cc:1261] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.325624 16266 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.326049 16286 tablet_copy_source_session.cc:205] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:44.327052 18908 consensus_queue.cc:1035] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:44.327514 19277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016fc1d28524a02bd762b74c6b08375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:44.328871 18189 raft_consensus.cc:2932] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.335482 15727 catalog_manager.cc:5484] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:44.336038 19301 tablet_copy_client.cc:523] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:44.339825 15853 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f25f3a86f0b494b89fe8f86aae4f4cd with delete_type TABLET_DATA_TOMBSTONED (TS 51b81a5ae56b4dd99af1b0167c1da1e2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35708
I20240501 20:10:44.343524 19067 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:44.344686 16265 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:44.350087 15733 catalog_manager.cc:5484] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:44.350409 16005 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:44.352093 19301 tablet_bootstrap.cc:492] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:44.357479 19337 tablet_replica.cc:330] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2: stopping tablet replica
I20240501 20:10:44.358268 19337 raft_consensus.cc:2229] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:44.360284 16133 raft_consensus.cc:1261] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.360561 19312 tablet_copy_client.cc:308] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:44.362387 18348 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:44.364236 19277 tablet_copy_client.cc:783] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:44.364454 19277 tablet_copy_client.cc:647] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:44.365829 18908 consensus_queue.cc:1035] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:44.366361 19277 tablet_copy_client.cc:523] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:44.372821 16133 raft_consensus.cc:2932] T f7fb9bc4cd73492da44bde1eecd627dc P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:44.379367 19337 raft_consensus.cc:2258] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:44.384536 15726 catalog_manager.cc:5484] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:44.378636 16266 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:44.387976 19277 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:44.390551 19337 ts_tablet_manager.cc:1875] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:44.398391 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:35897)
I20240501 20:10:44.398561 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:35897: session id = 047275bda1074ac5a63325e7e1109c61-d9a1a818ab7e44a5ac5c67f42e1dfe3b
I20240501 20:10:44.397933 16133 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:44.400365 19162 tablet_copy_client.cc:308] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:44.401499 15895 tablet_copy_source_session.cc:205] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:44.403040 19312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:44.423869 19312 tablet_copy_client.cc:783] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:44.426189 19312 tablet_copy_client.cc:647] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:44.428267 19312 tablet_copy_client.cc:523] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:44.432562 19312 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:44.433812 19337 ts_tablet_manager.cc:1888] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:44.434796 19337 log.cc:1196] T f7fb9bc4cd73492da44bde1eecd627dc P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7fb9bc4cd73492da44bde1eecd627dc
I20240501 20:10:44.435752 15719 catalog_manager.cc:4830] TS 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): tablet f7fb9bc4cd73492da44bde1eecd627dc (table rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]) successfully deleted
W20240501 20:10:44.438445 16072 consensus_peers.cc:485] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58ae613944b941e694f790e02cfe5292. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:44.440186 19337 tablet_replica.cc:330] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: stopping tablet replica
I20240501 20:10:44.443594 19337 raft_consensus.cc:2229] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:44.444094 19337 raft_consensus.cc:2258] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:44.446728 16286 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet eb90224d08f84a2faf7dcc063c6af56e from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:34279)
I20240501 20:10:44.446882 16286 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet eb90224d08f84a2faf7dcc063c6af56e from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:34279: session id = 047275bda1074ac5a63325e7e1109c61-eb90224d08f84a2faf7dcc063c6af56e
I20240501 20:10:44.447537 19337 ts_tablet_manager.cc:1875] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:44.449674 16286 tablet_copy_source_session.cc:205] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:44.451920 19162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb90224d08f84a2faf7dcc063c6af56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:44.458863 19162 tablet_copy_client.cc:783] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:44.459084 19162 tablet_copy_client.cc:647] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:44.461190 19162 tablet_copy_client.cc:523] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:44.464995 15873 consensus_queue.cc:237] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
W20240501 20:10:44.467705 15811 consensus_peers.cc:485] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9a1a818ab7e44a5ac5c67f42e1dfe3b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:44.468351 16133 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.468490 16004 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.469033 18852 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.469261 19198 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:10:44.470218 15809 consensus_peers.cc:485] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:44.472980 19162 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:44.473076 18852 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:44.476493 19337 ts_tablet_manager.cc:1888] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:44.477006 16134 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:44.485808 16004 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:44.487910 19337 log.cc:1196] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f25f3a86f0b494b89fe8f86aae4f4cd
I20240501 20:10:44.495608 15733 catalog_manager.cc:5484] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:44.498831 15719 catalog_manager.cc:4830] TS 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): tablet 0f25f3a86f0b494b89fe8f86aae4f4cd (table rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]) successfully deleted
I20240501 20:10:44.507493 18909 raft_consensus.cc:1050] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
I20240501 20:10:44.508261 18909 consensus_queue.cc:237] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:44.522861 18189 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.523352 19283 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.524056 16133 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.524523 19283 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.535511 16265 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:44.536018 19059 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:44.555529 19059 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.557436 19275 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:44.557860 19275 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:44.558125 19275 ts_tablet_manager.cc:1367] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.254s	user 0.041s	sys 0.004s
I20240501 20:10:44.558912 19275 raft_consensus.cc:348] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:44.559159 19275 raft_consensus.cc:729] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:44.559365 19275 consensus_queue.cc:260] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:44.559880 19275 ts_tablet_manager.cc:1398] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240501 20:10:44.562994 18189 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.565331 16133 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.568853 15733 catalog_manager.cc:5484] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 reported cstate change: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.576741 19301 tablet_bootstrap.cc:492] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:44.577129 19301 tablet_bootstrap.cc:492] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:44.577404 19301 ts_tablet_manager.cc:1367] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.225s	user 0.046s	sys 0.000s
I20240501 20:10:44.578181 19301 raft_consensus.cc:348] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:44.578441 19301 raft_consensus.cc:729] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:44.578683 19301 consensus_queue.cc:260] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:44.579828 19301 ts_tablet_manager.cc:1398] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240501 20:10:44.580439 16026 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session d3917663631a4479b15355381486591c-f3761f02c20a4448acd89f4a37f1193c received from {username='slave'} at 127.8.68.198:50675
I20240501 20:10:44.580591 16026 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session d3917663631a4479b15355381486591c-f3761f02c20a4448acd89f4a37f1193c on tablet f3761f02c20a4448acd89f4a37f1193c with peer d3917663631a4479b15355381486591c
I20240501 20:10:44.593158 15873 consensus_queue.cc:237] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:44.609984 16134 raft_consensus.cc:1261] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:44.610879 18852 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:44.611032 16286 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-c538581c71f646b5bc8d71cb6981f201 received from {username='slave'} at 127.8.68.199:39585
I20240501 20:10:44.611143 16286 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session cad00548f33849e7a19cecd760e1e429-c538581c71f646b5bc8d71cb6981f201 on tablet c538581c71f646b5bc8d71cb6981f201 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:44.611219 19312 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:44.611667 19312 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:44.611750 16265 raft_consensus.cc:1261] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:44.612169 19312 ts_tablet_manager.cc:1367] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.180s	user 0.044s	sys 0.005s
W20240501 20:10:44.612161 15808 consensus_peers.cc:485] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e483217b8761457d84edc49253877719. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:44.612905 19312 raft_consensus.cc:348] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:44.613155 19312 raft_consensus.cc:729] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:44.613346 19312 consensus_queue.cc:260] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:44.615048 19312 ts_tablet_manager.cc:1398] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:10:44.615024 18852 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:44.617040 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-d9a1a818ab7e44a5ac5c67f42e1dfe3b received from {username='slave'} at 127.8.68.200:35897
I20240501 20:10:44.617156 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-d9a1a818ab7e44a5ac5c67f42e1dfe3b on tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:44.621906 19277 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:44.622331 19277 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:44.622594 19277 ts_tablet_manager.cc:1367] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.235s	user 0.064s	sys 0.035s
I20240501 20:10:44.623332 19277 raft_consensus.cc:348] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:44.623611 19277 raft_consensus.cc:729] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:44.623824 19277 consensus_queue.cc:260] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:44.629205 16266 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.638219 16004 consensus_queue.cc:237] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:44.640276 18189 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.640713 18909 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:44.640832 16266 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:44.641320 18909 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:44.644222 18909 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.646857 18189 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.647212 18852 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:44.649219 16264 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:44.650897 16133 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:44.651402 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f904b62401ff4528b3fa65dd4e11512a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:44.653352 15733 catalog_manager.cc:5484] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 reported cstate change: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.654119 16266 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:44.657090 15726 catalog_manager.cc:5484] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:44.664428 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet f904b62401ff4528b3fa65dd4e11512a with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:44.680971 19401 tablet_replica.cc:330] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:44.681288 19401 raft_consensus.cc:2229] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:44.681603 19401 raft_consensus.cc:2258] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:44.691869 19401 ts_tablet_manager.cc:1875] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:44.713349 19401 ts_tablet_manager.cc:1888] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:44.727020 19275 ts_tablet_manager.cc:902] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:44.734380 16266 consensus_queue.cc:237] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:44.741147 19277 ts_tablet_manager.cc:1398] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.118s	user 0.011s	sys 0.037s
I20240501 20:10:44.741838 16286 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-b016fc1d28524a02bd762b74c6b08375 received from {username='slave'} at 127.8.68.197:55615
I20240501 20:10:44.741952 16286 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-b016fc1d28524a02bd762b74c6b08375 on tablet b016fc1d28524a02bd762b74c6b08375 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:44.750437 19401 log.cc:1196] T f904b62401ff4528b3fa65dd4e11512a P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/f904b62401ff4528b3fa65dd4e11512a
I20240501 20:10:44.751844 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet f904b62401ff4528b3fa65dd4e11512a (table rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]) successfully deleted
I20240501 20:10:44.771196 17927 raft_consensus.cc:1203] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:44.776381 19277 tablet_replica.cc:330] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: stopping tablet replica
I20240501 20:10:44.776944 19277 raft_consensus.cc:2229] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Raft consensus shutting down.
I20240501 20:10:44.777293 19277 raft_consensus.cc:2258] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Raft consensus is shut down!
I20240501 20:10:44.778846 19277 ts_tablet_manager.cc:1875] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:44.786232 19277 ts_tablet_manager.cc:1888] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:44.786417 19277 log.cc:1196] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/wals/b016fc1d28524a02bd762b74c6b08375
I20240501 20:10:44.786928 19277 ts_tablet_manager.cc:902] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:44.789556 19301 ts_tablet_manager.cc:902] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:44.794409 16134 raft_consensus.cc:1261] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:44.795301 16202 consensus_peers.cc:485] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3ba52a871714255aa1bd34d9daca920. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:44.795486 19159 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:44.801934 16004 raft_consensus.cc:1261] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:44.802642 19159 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:44.803211 19159 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:44.803658 16265 consensus_queue.cc:237] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:44.807955 16134 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:44.808393 19275 tablet_copy_client.cc:308] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:44.815404 15725 catalog_manager.cc:5484] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:44.818373 19312 ts_tablet_manager.cc:902] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:44.825088 16004 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:44.853341 19301 tablet_copy_client.cc:308] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:44.854082 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet c74a6da428b04df58b7edcdf88d81cca from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:60703)
I20240501 20:10:44.854243 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet c74a6da428b04df58b7edcdf88d81cca from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:60703: session id = d3917663631a4479b15355381486591c-c74a6da428b04df58b7edcdf88d81cca
I20240501 20:10:44.857309 15895 tablet_copy_source_session.cc:205] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:44.858690 19301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74a6da428b04df58b7edcdf88d81cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:44.865670 19301 tablet_copy_client.cc:783] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:44.865885 19301 tablet_copy_client.cc:647] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:44.867484 19301 tablet_copy_client.cc:523] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:44.871178 19301 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:44.881649 19210 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:44.882170 19210 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:44.882536 19210 ts_tablet_manager.cc:1367] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.994s	user 0.045s	sys 0.005s
I20240501 20:10:44.883437 19210 raft_consensus.cc:348] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:44.886739 19210 raft_consensus.cc:729] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:44.891857 16134 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:44.891984 16200 consensus_peers.cc:485] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98eb95ded28d433aad9523586e29947f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:44.892400 19320 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:44.894745 15872 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:44.895310 19320 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:44.903525 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:44.905740 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:44.917660 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet 58ae613944b941e694f790e02cfe5292 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:53283)
I20240501 20:10:44.917802 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet 58ae613944b941e694f790e02cfe5292 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:53283: session id = cad00548f33849e7a19cecd760e1e429-58ae613944b941e694f790e02cfe5292
I20240501 20:10:44.920732 16156 tablet_copy_source_session.cc:205] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:44.922297 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ae613944b941e694f790e02cfe5292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:44.928920 19275 tablet_copy_client.cc:783] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:44.929204 19275 tablet_copy_client.cc:647] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:44.930908 19275 tablet_copy_client.cc:523] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:44.931098 19320 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:44.934360 19275 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:44.935096 15875 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:44.938536 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:44.941144 15728 catalog_manager.cc:5484] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:44.892802 19277 tablet_copy_client.cc:308] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:44.963588 16286 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet b016fc1d28524a02bd762b74c6b08375 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:55615)
I20240501 20:10:44.963747 16286 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet b016fc1d28524a02bd762b74c6b08375 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:55615: session id = 9a0f0037700246bdb65e05aef95fb2be-b016fc1d28524a02bd762b74c6b08375
I20240501 20:10:44.966647 16286 tablet_copy_source_session.cc:205] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:44.967739 19277 ts_tablet_manager.cc:1875] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:10:44.973462 18060 raft_consensus.cc:1203] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:44.973824 19277 ts_tablet_manager.cc:1888] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:10:44.974084 19277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016fc1d28524a02bd762b74c6b08375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:44.977995 19277 tablet_copy_client.cc:783] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:44.978206 19277 tablet_copy_client.cc:647] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:44.980263 19210 consensus_queue.cc:260] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:44.982384 19277 tablet_copy_client.cc:523] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:44.987053 19210 ts_tablet_manager.cc:1398] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.104s	user 0.002s	sys 0.000s
I20240501 20:10:44.987120 19301 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:44.987598 19301 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:44.987702 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-36465c82164a4765a63590cfd5d413a4 received from {username='slave'} at 127.8.68.197:53423
I20240501 20:10:44.987876 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-36465c82164a4765a63590cfd5d413a4 on tablet 36465c82164a4765a63590cfd5d413a4 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:44.987872 19301 ts_tablet_manager.cc:1367] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.117s	user 0.056s	sys 0.000s
I20240501 20:10:44.988724 19301 raft_consensus.cc:348] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:44.988981 19301 raft_consensus.cc:729] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:44.989240 19301 consensus_queue.cc:260] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:44.990842 19277 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:44.993898 19301 ts_tablet_manager.cc:1398] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 20:10:44.996127 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session d3917663631a4479b15355381486591c-c74a6da428b04df58b7edcdf88d81cca received from {username='slave'} at 127.8.68.198:60703
I20240501 20:10:44.996273 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session d3917663631a4479b15355381486591c-c74a6da428b04df58b7edcdf88d81cca on tablet c74a6da428b04df58b7edcdf88d81cca with peer d3917663631a4479b15355381486591c
I20240501 20:10:44.997437 19210 ts_tablet_manager.cc:902] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:45.009519 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:45.011808 19312 tablet_copy_client.cc:308] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:45.012816 16286 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet b016fc1d28524a02bd762b74c6b08375 from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:34279)
I20240501 20:10:45.012986 16286 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet b016fc1d28524a02bd762b74c6b08375 from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:34279: session id = 047275bda1074ac5a63325e7e1109c61-b016fc1d28524a02bd762b74c6b08375
I20240501 20:10:45.015993 16286 tablet_copy_source_session.cc:205] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:45.028592 19312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016fc1d28524a02bd762b74c6b08375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:45.035750 18189 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:45.059798 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:45.067791 16133 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:45.069833 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:45.092008 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:45.104887 19301 ts_tablet_manager.cc:902] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:45.135529 19301 tablet_copy_client.cc:308] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:45.136586 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet e483217b8761457d84edc49253877719 from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:60703)
I20240501 20:10:45.136786 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet e483217b8761457d84edc49253877719 from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:60703: session id = d3917663631a4479b15355381486591c-e483217b8761457d84edc49253877719
I20240501 20:10:45.139611 15895 tablet_copy_source_session.cc:205] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:45.145273 19301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e483217b8761457d84edc49253877719. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:45.153617 19301 tablet_copy_client.cc:783] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:45.153858 19301 tablet_copy_client.cc:647] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:45.155514 19301 tablet_copy_client.cc:523] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:45.158766 19301 tablet_bootstrap.cc:492] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:45.185221 16265 consensus_queue.cc:237] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:45.186100 19277 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:45.186548 19277 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:45.186808 19277 ts_tablet_manager.cc:1367] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.196s	user 0.054s	sys 0.049s
I20240501 20:10:45.189177 16004 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.189785 19320 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:10:45.193645 16202 consensus_peers.cc:485] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:45.205827 19031 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:45.206634 16004 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:45.214601 19277 raft_consensus.cc:348] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:45.214865 19277 raft_consensus.cc:729] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:45.215129 19277 consensus_queue.cc:260] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:45.215014 15728 catalog_manager.cc:5484] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 5 to 6, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:45.215698 19277 ts_tablet_manager.cc:1398] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.029s	user 0.001s	sys 0.001s
I20240501 20:10:45.216886 16286 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-b016fc1d28524a02bd762b74c6b08375 received from {username='slave'} at 127.8.68.197:55615
I20240501 20:10:45.217010 16286 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-b016fc1d28524a02bd762b74c6b08375 on tablet b016fc1d28524a02bd762b74c6b08375 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:45.226893 19210 tablet_copy_client.cc:308] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:45.227700 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:49547)
I20240501 20:10:45.227885 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:49547: session id = 9a0f0037700246bdb65e05aef95fb2be-d9a1a818ab7e44a5ac5c67f42e1dfe3b
I20240501 20:10:45.231076 15895 tablet_copy_source_session.cc:205] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:45.238214 19312 tablet_copy_client.cc:783] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:45.238615 19312 tablet_copy_client.cc:647] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:45.252617 15872 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.254092 19031 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:10:45.272294 16200 consensus_peers.cc:485] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:45.274952 15872 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:45.277050 19162 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:45.277474 19162 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:45.277753 19162 ts_tablet_manager.cc:1367] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.805s	user 0.040s	sys 0.006s
I20240501 20:10:45.278869 19301 tablet_bootstrap.cc:492] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:45.278707 19162 raft_consensus.cc:348] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:45.278971 19162 raft_consensus.cc:729] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:45.279316 19301 tablet_bootstrap.cc:492] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:45.279206 19162 consensus_queue.cc:260] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:45.279587 19301 ts_tablet_manager.cc:1367] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.121s	user 0.059s	sys 0.013s
I20240501 20:10:45.280086 19033 tablet_bootstrap.cc:492] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:45.280354 19301 raft_consensus.cc:348] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:45.280583 19033 tablet_bootstrap.cc:492] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:45.280622 19301 raft_consensus.cc:729] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:45.280867 19033 ts_tablet_manager.cc:1367] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.981s	user 0.051s	sys 0.000s
I20240501 20:10:45.280829 19301 consensus_queue.cc:260] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:45.281417 19301 ts_tablet_manager.cc:1398] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:45.282027 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session d3917663631a4479b15355381486591c-e483217b8761457d84edc49253877719 received from {username='slave'} at 127.8.68.198:60703
I20240501 20:10:45.282177 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session d3917663631a4479b15355381486591c-e483217b8761457d84edc49253877719 on tablet e483217b8761457d84edc49253877719 with peer d3917663631a4479b15355381486591c
I20240501 20:10:45.285720 19033 raft_consensus.cc:348] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:45.285928 19033 raft_consensus.cc:729] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:45.286123 19033 consensus_queue.cc:260] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:45.289887 19033 ts_tablet_manager.cc:1398] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240501 20:10:45.290768 16026 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session d3917663631a4479b15355381486591c-67d59d1654ef4ab3ac996f55a73b313f received from {username='slave'} at 127.8.68.198:50675
I20240501 20:10:45.290932 16026 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session d3917663631a4479b15355381486591c-67d59d1654ef4ab3ac996f55a73b313f on tablet 67d59d1654ef4ab3ac996f55a73b313f with peer d3917663631a4479b15355381486591c
I20240501 20:10:45.291126 19162 ts_tablet_manager.cc:1398] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.000s
I20240501 20:10:45.294855 18909 raft_consensus.cc:1050] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:45.295817 18909 consensus_queue.cc:237] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:45.310382 19275 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:45.310804 19275 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:45.312350 19275 ts_tablet_manager.cc:1367] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.378s	user 0.046s	sys 0.010s
I20240501 20:10:45.313285 19275 raft_consensus.cc:348] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:45.313443 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:45.313521 19275 raft_consensus.cc:729] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:45.313735 19275 consensus_queue.cc:260] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:45.314371 19275 ts_tablet_manager.cc:1398] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:45.315673 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-58ae613944b941e694f790e02cfe5292 received from {username='slave'} at 127.8.68.199:53283
I20240501 20:10:45.315838 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session cad00548f33849e7a19cecd760e1e429-58ae613944b941e694f790e02cfe5292 on tablet 58ae613944b941e694f790e02cfe5292 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:45.331887 19210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:45.341631 17927 raft_consensus.cc:1203] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:45.346206 16286 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-eb90224d08f84a2faf7dcc063c6af56e received from {username='slave'} at 127.8.68.200:34279
I20240501 20:10:45.346365 16286 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-eb90224d08f84a2faf7dcc063c6af56e on tablet eb90224d08f84a2faf7dcc063c6af56e with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:45.347415 19312 tablet_copy_client.cc:523] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:45.351346 19312 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:45.354266 16265 raft_consensus.cc:1261] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.355038 17927 raft_consensus.cc:1261] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.355154 15875 raft_consensus.cc:1261] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.355722 19387 consensus_queue.cc:1035] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.356303 19387 consensus_queue.cc:1035] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.356691 19059 consensus_queue.cc:1035] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.358198 15872 consensus_queue.cc:237] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:45.363121 17793 raft_consensus.cc:1203] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240501 20:10:45.372081 15809 consensus_peers.cc:485] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddf5ddac08364043bf2e27184d5d3213. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:45.373601 16265 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:45.374220 18852 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:45.377779 19275 ts_tablet_manager.cc:902] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:45.379060 19162 ts_tablet_manager.cc:902] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:45.384513 18852 raft_consensus.cc:1050] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
I20240501 20:10:45.385434 18852 consensus_queue.cc:237] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:45.386549 19210 tablet_copy_client.cc:783] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:45.386832 19210 tablet_copy_client.cc:647] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:45.387929 18189 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.388661 19198 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:10:45.389163 15811 consensus_peers.cc:485] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:45.389516 19210 tablet_copy_client.cc:523] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:45.393864 19210 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
W20240501 20:10:45.405789 16199 consensus_peers.cc:485] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b016fc1d28524a02bd762b74c6b08375. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:45.410375 19198 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:45.414832 16265 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:45.419994 15733 catalog_manager.cc:5484] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
W20240501 20:10:45.427670 15939 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:44869, user_credentials={real_user=slave}} blocked reactor thread for 71914.9us
I20240501 20:10:45.435125 19320 raft_consensus.cc:1050] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:45.435930 19320 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:45.438642 16004 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:45.439988 19522 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:45.450502 16004 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:45.461978 16134 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.462576 19522 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:45.464455 16004 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.465530 19522 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:45.465919 19522 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:45.469518 16134 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:45.475914 15727 catalog_manager.cc:5484] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:45.477864 16004 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:45.492660 18060 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.493213 19159 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.494655 19511 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.501086 16134 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.501730 19159 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.502396 15727 catalog_manager.cc:5484] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:45.505043 17929 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.513357 16004 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.513856 19159 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.525632 18189 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:45.527849 19312 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:45.528323 19312 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:45.528604 19312 ts_tablet_manager.cc:1367] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.177s	user 0.078s	sys 0.021s
I20240501 20:10:45.529450 19312 raft_consensus.cc:348] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:45.529709 19312 raft_consensus.cc:729] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:45.539629 19159 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:45.540465 17927 raft_consensus.cc:1203] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:45.544610 17929 raft_consensus.cc:1203] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:45.547348 18059 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:45.550148 19312 consensus_queue.cc:260] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:45.551194 15733 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:45.552237 19312 ts_tablet_manager.cc:1398] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.024s	user 0.003s	sys 0.000s
I20240501 20:10:45.552373 16005 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:45.560071 16133 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:45.564962 16266 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:45.567009 16133 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.567585 19320 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:45.569281 18060 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.569885 19320 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:45.572711 19159 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:45.578112 16133 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:45.579864 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c538581c71f646b5bc8d71cb6981f201 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:45.582610 15733 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:45.583896 18059 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:45.617609 18060 raft_consensus.cc:1203] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240501 20:10:45.631065 15810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:46677, user_credentials={real_user=slave}} blocked reactor thread for 148270us
I20240501 20:10:45.632936 15875 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.636344 19275 tablet_copy_client.cc:308] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:45.637913 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet c538581c71f646b5bc8d71cb6981f201 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:45.637945 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet d3ba52a871714255aa1bd34d9daca920 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:39585)
I20240501 20:10:45.638195 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet d3ba52a871714255aa1bd34d9daca920 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:39585: session id = cad00548f33849e7a19cecd760e1e429-d3ba52a871714255aa1bd34d9daca920
I20240501 20:10:45.638912 16286 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-b016fc1d28524a02bd762b74c6b08375 received from {username='slave'} at 127.8.68.200:34279
I20240501 20:10:45.639046 16286 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-b016fc1d28524a02bd762b74c6b08375 on tablet b016fc1d28524a02bd762b74c6b08375 with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:45.640683 19560 tablet_replica.cc:330] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:45.641353 16285 tablet_copy_source_session.cc:205] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:45.641549 19560 raft_consensus.cc:2229] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:45.642827 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ba52a871714255aa1bd34d9daca920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:45.643677 19560 raft_consensus.cc:2258] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:45.644208 19162 tablet_copy_client.cc:308] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:45.644896 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet 98eb95ded28d433aad9523586e29947f from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:34279)
I20240501 20:10:45.645177 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet 98eb95ded28d433aad9523586e29947f from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:34279: session id = 047275bda1074ac5a63325e7e1109c61-98eb95ded28d433aad9523586e29947f
I20240501 20:10:45.645890 16004 consensus_queue.cc:237] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:45.646097 16265 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.648452 16285 tablet_copy_source_session.cc:205] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:45.651139 19560 ts_tablet_manager.cc:1875] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:45.654774 17927 raft_consensus.cc:1261] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.655715 19511 consensus_queue.cc:1035] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:45.658033 16266 raft_consensus.cc:1261] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.658841 19162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98eb95ded28d433aad9523586e29947f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:45.660167 19511 consensus_queue.cc:1035] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:45.662361 19387 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.663159 16266 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.664482 19560 ts_tablet_manager.cc:1888] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:45.664621 19560 log.cc:1196] T c538581c71f646b5bc8d71cb6981f201 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/c538581c71f646b5bc8d71cb6981f201
I20240501 20:10:45.667150 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet c538581c71f646b5bc8d71cb6981f201 (table rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]) successfully deleted
I20240501 20:10:45.668476 15728 catalog_manager.cc:5484] T f3761f02c20a4448acd89f4a37f1193c P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.671355 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3761f02c20a4448acd89f4a37f1193c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:45.673197 15854 tablet_service.cc:1514] Processing DeleteTablet for tablet f3761f02c20a4448acd89f4a37f1193c with delete_type TABLET_DATA_TOMBSTONED (TS 51b81a5ae56b4dd99af1b0167c1da1e2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35708
I20240501 20:10:45.683529 19566 tablet_replica.cc:330] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2: stopping tablet replica
I20240501 20:10:45.685225 19566 raft_consensus.cc:2229] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:45.685599 19566 raft_consensus.cc:2258] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:45.686900 19566 ts_tablet_manager.cc:1875] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:45.690009 17927 raft_consensus.cc:2932] T f3761f02c20a4448acd89f4a37f1193c P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.703940 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:45.721093 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:45.734495 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:45.734647 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:45.746749 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:45.758697 18189 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:45.762120 19275 tablet_copy_client.cc:783] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:45.762404 19275 tablet_copy_client.cc:647] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:45.762607 19162 tablet_copy_client.cc:783] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:45.762866 19162 tablet_copy_client.cc:647] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:45.767462 19275 tablet_copy_client.cc:523] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:45.770491 18815 raft_consensus.cc:1050] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:45.771368 18815 consensus_queue.cc:237] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:45.772322 19162 tablet_copy_client.cc:523] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:45.775928 19162 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:45.776674 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:45.777292 18190 raft_consensus.cc:1203] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:45.786976 19275 tablet_bootstrap.cc:492] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:45.794675 19033 ts_tablet_manager.cc:902] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:45.801259 19536 raft_consensus.cc:1050] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:45.802042 19536 consensus_queue.cc:237] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:45.810001 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:45.810602 17793 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:45.823439 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:45.825490 19566 ts_tablet_manager.cc:1888] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:45.825685 19566 log.cc:1196] T f3761f02c20a4448acd89f4a37f1193c P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/f3761f02c20a4448acd89f4a37f1193c
I20240501 20:10:45.826625 15719 catalog_manager.cc:4830] TS 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): tablet f3761f02c20a4448acd89f4a37f1193c (table rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]) successfully deleted
I20240501 20:10:45.832190 17793 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.832881 19067 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.833855 16266 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.834393 19067 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.834990 15875 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.835546 19067 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.871511 19070 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:45.872140 15875 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:45.872524 17927 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.873134 19522 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.876276 16133 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.876946 17793 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:45.877525 16265 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:45.878614 19536 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.881404 19312 ts_tablet_manager.cc:902] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:45.881286 15733 catalog_manager.cc:5484] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:45.884626 16266 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:45.885212 18852 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:45.906525 16134 consensus_queue.cc:237] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:45.909987 15875 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.910835 18815 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:45.912954 17795 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:45.913544 16266 consensus_queue.cc:237] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:45.914314 19067 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:45.914824 19067 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:45.918890 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 36465c82164a4765a63590cfd5d413a4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:45.919893 17793 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:45.921020 15875 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:45.924019 15728 catalog_manager.cc:5484] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:45.928457 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet 36465c82164a4765a63590cfd5d413a4 with delete_type TABLET_DATA_TOMBSTONED (TS f856e0d1074b431eb19763847958b89e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44866
I20240501 20:10:45.939877 19033 tablet_copy_client.cc:308] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:45.944792 19597 tablet_replica.cc:330] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e: stopping tablet replica
I20240501 20:10:45.946228 19597 raft_consensus.cc:2229] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:45.948968 16134 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:45.949854 15875 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:45.948400 16202 consensus_peers.cc:485] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1474070bbdf147d2ae3313fdf9d71c19. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:45.949472 19031 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:45.952977 19159 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:45.952999 19597 raft_consensus.cc:2258] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:45.963102 19597 ts_tablet_manager.cc:1875] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:45.968501 19522 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.974751 17927 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.979382 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet b016fc1d28524a02bd762b74c6b08375 from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:46329)
I20240501 20:10:45.979542 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet b016fc1d28524a02bd762b74c6b08375 from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:46329: session id = d3917663631a4479b15355381486591c-b016fc1d28524a02bd762b74c6b08375
I20240501 20:10:45.982729 16285 tablet_copy_source_session.cc:205] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:45.984359 19033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b016fc1d28524a02bd762b74c6b08375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:45.985571 16265 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:45.988173 15727 catalog_manager.cc:5484] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:46.000325 15872 consensus_queue.cc:237] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:46.003000 18189 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.004092 16136 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.004364 19033 tablet_copy_client.cc:783] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:46.004598 19033 tablet_copy_client.cc:647] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:46.006165 19519 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:46.006706 19519 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:46.011094 19519 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.015632 18189 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.019963 16005 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.020606 18852 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:46.022004 16005 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.023521 19033 tablet_copy_client.cc:523] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:46.026919 19033 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:46.035295 18852 raft_consensus.cc:1050] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:46.036087 18852 consensus_queue.cc:237] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:46.036990 19159 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:46.043064 16136 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:46.044893 15733 catalog_manager.cc:5484] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 reported cstate change: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.045439 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.046970 16135 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.048885 15872 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:46.050009 15727 catalog_manager.cc:5484] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:46.060894 16134 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.067447 15872 consensus_queue.cc:237] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:46.069643 18815 raft_consensus.cc:1050] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:46.070446 18815 consensus_queue.cc:237] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:46.073024 15875 consensus_queue.cc:237] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:46.073248 17927 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.074314 19522 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:46.076215 17929 raft_consensus.cc:1261] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.076771 19198 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.077279 17929 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.079186 19531 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:10:46.081904 15726 catalog_manager.cc:5484] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c reported cstate change: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.082432 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c74a6da428b04df58b7edcdf88d81cca with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.082710 19312 tablet_copy_client.cc:308] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:46.084093 16135 raft_consensus.cc:1261] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.084884 19531 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.085453 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet c74a6da428b04df58b7edcdf88d81cca with delete_type TABLET_DATA_TOMBSTONED (TS f856e0d1074b431eb19763847958b89e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44866
I20240501 20:10:46.085944 16005 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:46.087198 19536 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:46.089203 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b with delete_type TABLET_DATA_TOMBSTONED (TS 9a0f0037700246bdb65e05aef95fb2be not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53350
I20240501 20:10:46.091346 19522 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.095109 16005 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.111799 16265 raft_consensus.cc:1261] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.114475 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.118518 15726 catalog_manager.cc:5484] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 7 to 8, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
W20240501 20:10:46.120177 15721 catalog_manager.cc:4810] TS 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): delete failed for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b already in progress: copying tablet
I20240501 20:10:46.122136 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:46.124627 19387 raft_consensus.cc:1050] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:46.125460 19387 consensus_queue.cc:237] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:46.139693 19634 tablet_replica.cc:330] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:46.140599 19634 raft_consensus.cc:2229] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:46.141047 19634 raft_consensus.cc:2258] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:46.141785 16133 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.142423 19522 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:46.144248 16133 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.083765 18189 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:46.151005 19522 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:46.153100 18189 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.163918 18060 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.164588 19070 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.174849 19634 ts_tablet_manager.cc:1875] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.177263 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet ddf5ddac08364043bf2e27184d5d3213 from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:35897)
I20240501 20:10:46.177443 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet ddf5ddac08364043bf2e27184d5d3213 from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:35897: session id = 047275bda1074ac5a63325e7e1109c61-ddf5ddac08364043bf2e27184d5d3213
I20240501 20:10:46.176584 19522 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.180616 15895 tablet_copy_source_session.cc:205] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:46.181675 19519 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.185635 17927 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.189126 16005 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.189775 19070 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.190629 19275 tablet_bootstrap.cc:492] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:46.191020 19275 tablet_bootstrap.cc:492] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:46.191939 19275 ts_tablet_manager.cc:1367] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.405s	user 0.036s	sys 0.012s
I20240501 20:10:46.192672 19275 raft_consensus.cc:348] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:46.192917 19275 raft_consensus.cc:729] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:46.193310 16133 raft_consensus.cc:1261] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.193814 17928 raft_consensus.cc:1261] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.194267 15875 raft_consensus.cc:1261] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.195087 19059 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.195498 18909 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.196259 19059 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.200227 16265 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.201011 19070 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.207316 19275 consensus_queue.cc:260] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:46.218737 19312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf5ddac08364043bf2e27184d5d3213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:46.222172 15728 catalog_manager.cc:5484] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c reported cstate change: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.232206 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:46.233673 19070 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:46.235316 16005 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:46.237154 19033 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:46.237573 19033 tablet_bootstrap.cc:492] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:46.239491 16263 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:46.240703 15728 catalog_manager.cc:5484] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:46.249125 19033 ts_tablet_manager.cc:1367] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.222s	user 0.039s	sys 0.011s
I20240501 20:10:46.252763 19059 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.252745 16266 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.257270 17928 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.263698 15727 catalog_manager.cc:5484] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:46.264469 16134 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.270577 16135 consensus_queue.cc:237] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:46.272806 16005 consensus_queue.cc:237] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:46.274008 16265 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.274628 19067 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:46.281149 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:46.281355 19275 ts_tablet_manager.cc:1398] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.089s	user 0.007s	sys 0.007s
I20240501 20:10:46.282081 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-d3ba52a871714255aa1bd34d9daca920 received from {username='slave'} at 127.8.68.199:39585
I20240501 20:10:46.282231 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session cad00548f33849e7a19cecd760e1e429-d3ba52a871714255aa1bd34d9daca920 on tablet d3ba52a871714255aa1bd34d9daca920 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:46.251518 19033 raft_consensus.cc:348] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:46.284508 19033 raft_consensus.cc:729] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:46.284737 19033 consensus_queue.cc:260] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:46.286008 19597 ts_tablet_manager.cc:1888] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:46.286079 19033 ts_tablet_manager.cc:1398] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.037s	user 0.002s	sys 0.000s
W20240501 20:10:46.288249 15808 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:44869, user_credentials={real_user=slave}} blocked reactor thread for 58364.9us
I20240501 20:10:46.295516 19070 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:46.299714 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 58ae613944b941e694f790e02cfe5292 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.307158 15875 consensus_queue.cc:237] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:46.307703 15725 catalog_manager.cc:5484] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:46.311895 17928 raft_consensus.cc:1261] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.312510 19623 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:46.315935 19519 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.320237 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e483217b8761457d84edc49253877719 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.320549 18059 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.321262 18815 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.321987 18060 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:46.322255 15725 catalog_manager.cc:5484] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:46.324292 17928 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.284866 16135 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.330318 18059 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:46.312374 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet 58ae613944b941e694f790e02cfe5292 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:46.341702 15875 consensus_queue.cc:237] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:46.301842 16265 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:46.343817 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet e483217b8761457d84edc49253877719 with delete_type TABLET_DATA_TOMBSTONED (TS f856e0d1074b431eb19763847958b89e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44866
I20240501 20:10:46.352483 19597 log.cc:1196] T 36465c82164a4765a63590cfd5d413a4 P f856e0d1074b431eb19763847958b89e: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/36465c82164a4765a63590cfd5d413a4
I20240501 20:10:46.353910 19597 tablet_replica.cc:330] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e: stopping tablet replica
I20240501 20:10:46.354136 19597 raft_consensus.cc:2229] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:46.345085 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session d3917663631a4479b15355381486591c-b016fc1d28524a02bd762b74c6b08375 received from {username='slave'} at 127.8.68.198:46329
I20240501 20:10:46.354427 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session d3917663631a4479b15355381486591c-b016fc1d28524a02bd762b74c6b08375 on tablet b016fc1d28524a02bd762b74c6b08375 with peer d3917663631a4479b15355381486591c
I20240501 20:10:46.355086 15718 catalog_manager.cc:4830] TS f856e0d1074b431eb19763847958b89e (127.8.68.196:36919): tablet 36465c82164a4765a63590cfd5d413a4 (table rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]) successfully deleted
I20240501 20:10:46.357972 19496 raft_consensus.cc:1050] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:46.359138 19496 consensus_queue.cc:237] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:46.370862 17928 raft_consensus.cc:1261] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.371227 19275 ts_tablet_manager.cc:902] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:46.371771 16136 raft_consensus.cc:1261] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.371996 19387 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:46.372444 19059 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:46.374255 15985 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:46.375041 17928 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.377681 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:46.374531 19059 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.382202 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67d59d1654ef4ab3ac996f55a73b313f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.385349 19634 ts_tablet_manager.cc:1888] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 20:10:46.393185 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:46.394690 19661 raft_consensus.cc:1050] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
I20240501 20:10:46.395519 19661 consensus_queue.cc:237] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:46.401062 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:46.405153 17928 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.406189 17793 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.406327 19664 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:46.407029 19662 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:46.409538 18059 raft_consensus.cc:1203] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:46.410039 16136 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.413784 19597 raft_consensus.cc:2258] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:46.414213 16004 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.414880 19597 ts_tablet_manager.cc:1875] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.414904 19663 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:46.418839 19659 tablet_replica.cc:330] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:46.419064 19659 raft_consensus.cc:2229] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:46.419309 19659 raft_consensus.cc:2258] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:46.384830 15726 catalog_manager.cc:5484] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 5 to 6, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:46.431885 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:46.434998 19634 log.cc:1196] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/d9a1a818ab7e44a5ac5c67f42e1dfe3b
I20240501 20:10:46.440310 16134 raft_consensus.cc:1261] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.441008 18188 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.441792 18190 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.442023 15874 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.442343 15873 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.442517 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b (table rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]) successfully deleted
I20240501 20:10:46.443603 19664 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.444310 19664 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:10:46.446795 16072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.199:44275, user_credentials={real_user=slave}} blocked reactor thread for 108447us
I20240501 20:10:46.447927 19664 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.453598 19664 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:46.456656 19159 raft_consensus.cc:1013] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: attempt to promote peer 047275bda1074ac5a63325e7e1109c61: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:10:46.457180 15854 tablet_service.cc:1514] Processing DeleteTablet for tablet 67d59d1654ef4ab3ac996f55a73b313f with delete_type TABLET_DATA_TOMBSTONED (TS 51b81a5ae56b4dd99af1b0167c1da1e2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35708
I20240501 20:10:46.459972 19659 ts_tablet_manager.cc:1875] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.460994 19312 tablet_copy_client.cc:783] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:46.461287 19312 tablet_copy_client.cc:647] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:46.467553 19661 raft_consensus.cc:1013] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: attempt to promote peer d3917663631a4479b15355381486591c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:10:46.469321 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:46.464725 19162 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:46.474920 19162 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:46.475239 19162 ts_tablet_manager.cc:1367] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.699s	user 0.042s	sys 0.006s
I20240501 20:10:46.476059 19162 raft_consensus.cc:348] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:46.476384 19162 raft_consensus.cc:729] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:46.476573 19162 consensus_queue.cc:260] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:46.478140 19162 ts_tablet_manager.cc:1398] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 20:10:46.478737 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-98eb95ded28d433aad9523586e29947f received from {username='slave'} at 127.8.68.200:34279
I20240501 20:10:46.478871 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-98eb95ded28d433aad9523586e29947f on tablet 98eb95ded28d433aad9523586e29947f with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:46.489123 19312 tablet_copy_client.cc:523] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:46.492866 19312 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:46.508893 15874 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.513758 19661 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.520069 16005 raft_consensus.cc:1261] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:46.520371 15809 consensus_peers.cc:485] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d48de8e0c12b432ea5079390400302c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.520577 16266 raft_consensus.cc:1261] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:46.520771 19522 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:46.521241 18852 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:46.522830 15726 catalog_manager.cc:5484] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.543727 19519 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:46.547255 16004 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.550026 15726 catalog_manager.cc:5484] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:46.552019 17793 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.552067 18190 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.554589 17928 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.560170 16265 consensus_queue.cc:237] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:46.563586 19679 tablet_replica.cc:330] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: stopping tablet replica
I20240501 20:10:46.563855 19679 raft_consensus.cc:2229] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:46.564136 19679 pending_rounds.cc:62] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: Trying to abort 1 pending ops.
I20240501 20:10:46.564208 17927 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:46.564304 19679 pending_rounds.cc:68] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022978032127586304 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "67d59d1654ef4ab3ac996f55a73b313f" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } } }
I20240501 20:10:46.564945 19679 raft_consensus.cc:2894] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:10:46.564924 19031 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:46.565114 19679 raft_consensus.cc:2866] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:46.565280 19679 raft_consensus.cc:2258] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:46.565984 19597 ts_tablet_manager.cc:1888] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:46.567550 15875 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:46.568671 15874 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.572844 19275 tablet_copy_client.cc:308] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:46.577628 19210 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:46.578105 19210 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:46.568096 19320 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.568578 19159 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:46.579121 15726 catalog_manager.cc:5484] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.568784 19660 raft_consensus.cc:1013] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: attempt to promote peer d3917663631a4479b15355381486591c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:10:46.583473 19679 ts_tablet_manager.cc:1875] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.584017 18190 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.572234 16263 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:46.573972 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet 1474070bbdf147d2ae3313fdf9d71c19 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:39585)
I20240501 20:10:46.585170 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet 1474070bbdf147d2ae3313fdf9d71c19 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:39585: session id = cad00548f33849e7a19cecd760e1e429-1474070bbdf147d2ae3313fdf9d71c19
I20240501 20:10:46.588331 16285 tablet_copy_source_session.cc:205] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:46.588845 19597 log.cc:1196] T c74a6da428b04df58b7edcdf88d81cca P f856e0d1074b431eb19763847958b89e: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/c74a6da428b04df58b7edcdf88d81cca
I20240501 20:10:46.589679 15718 catalog_manager.cc:4830] TS f856e0d1074b431eb19763847958b89e (127.8.68.196:36919): tablet c74a6da428b04df58b7edcdf88d81cca (table rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]) successfully deleted
I20240501 20:10:46.589872 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1474070bbdf147d2ae3313fdf9d71c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:46.589919 19597 tablet_replica.cc:330] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e: stopping tablet replica
I20240501 20:10:46.590227 19597 raft_consensus.cc:2229] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:46.590349 17795 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:46.590508 19597 raft_consensus.cc:2258] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:46.591606 19031 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:46.591782 19597 ts_tablet_manager.cc:1875] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.593394 16003 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:10:46.521451 16069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:36919, user_credentials={real_user=slave}} blocked reactor thread for 83654.4us
I20240501 20:10:46.597419 16133 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.603411 19670 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:46.605218 19670 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.606348 19659 ts_tablet_manager.cc:1888] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:46.606478 19659 log.cc:1196] T 58ae613944b941e694f790e02cfe5292 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/58ae613944b941e694f790e02cfe5292
I20240501 20:10:46.607237 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet 58ae613944b941e694f790e02cfe5292 (table rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]) successfully deleted
I20240501 20:10:46.607369 16003 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.612110 19210 ts_tablet_manager.cc:1367] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 1.218s	user 0.054s	sys 0.002s
I20240501 20:10:46.612967 19210 raft_consensus.cc:348] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:46.613240 19210 raft_consensus.cc:729] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:46.613531 19210 consensus_queue.cc:260] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:46.613692 15728 catalog_manager.cc:5484] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 7 to 8, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.614171 19210 ts_tablet_manager.cc:1398] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:46.615449 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-d9a1a818ab7e44a5ac5c67f42e1dfe3b received from {username='slave'} at 127.8.68.197:49547
I20240501 20:10:46.615581 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-d9a1a818ab7e44a5ac5c67f42e1dfe3b on tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:46.618072 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:53350
I20240501 20:10:46.618144 18170 tablet_service.cc:1514] Processing DeleteTablet for tablet b016fc1d28524a02bd762b74c6b08375 with delete_type TABLET_DATA_TOMBSTONED (TS 047275bda1074ac5a63325e7e1109c61 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35670
I20240501 20:10:46.618455 19631 tablet_replica.cc:330] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: stopping tablet replica
I20240501 20:10:46.618620 19631 raft_consensus.cc:2229] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Raft consensus shutting down.
I20240501 20:10:46.618819 19631 raft_consensus.cc:2258] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Raft consensus is shut down!
I20240501 20:10:46.619563 19631 ts_tablet_manager.cc:1875] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.626157 19631 ts_tablet_manager.cc:1888] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:46.626356 19631 log.cc:1196] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/wals/d9a1a818ab7e44a5ac5c67f42e1dfe3b
I20240501 20:10:46.627226 15721 catalog_manager.cc:4830] TS 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b (table rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]) successfully deleted
I20240501 20:10:46.654186 18169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:46.656504 17795 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.657068 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:46.657127 19522 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:46.664625 19679 ts_tablet_manager.cc:1888] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:46.665853 16005 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:46.666839 19679 log.cc:1196] T 67d59d1654ef4ab3ac996f55a73b313f P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/67d59d1654ef4ab3ac996f55a73b313f
I20240501 20:10:46.668733 15719 catalog_manager.cc:4830] TS 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): tablet 67d59d1654ef4ab3ac996f55a73b313f (table rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]) successfully deleted
I20240501 20:10:46.687181 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:46.690780 16133 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.691463 19697 tablet_replica.cc:330] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: stopping tablet replica
I20240501 20:10:46.692145 19697 raft_consensus.cc:2229] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 20:10:46.692363 19275 tablet_copy_client.cc:783] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:46.692791 19275 tablet_copy_client.cc:647] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:46.606472 19320 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.705524 16265 consensus_queue.cc:237] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:46.708575 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b016fc1d28524a02bd762b74c6b08375 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.708639 17928 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.710134 15873 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.714409 19662 raft_consensus.cc:1050] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:46.715247 19662 consensus_queue.cc:237] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:46.716548 19275 tablet_copy_client.cc:523] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:46.718545 17928 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:46.718778 16003 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:46.718839 17793 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:46.719137 19320 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:46.719810 19031 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:46.720961 15873 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:46.721489 19496 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:46.722000 19320 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:46.723359 19496 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.724411 16003 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.728940 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:46.731854 17795 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.734506 19697 raft_consensus.cc:2258] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 20:10:46.736124 19697 ts_tablet_manager.cc:1875] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.737282 15726 catalog_manager.cc:5484] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 8 to 9, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.740319 15875 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.741149 15874 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.745436 19663 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:46.748009 15728 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet b016fc1d28524a02bd762b74c6b08375 with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8
I20240501 20:10:46.750219 19663 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.752930 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:46.753741 19275 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:46.753862 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb90224d08f84a2faf7dcc063c6af56e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.755896 15875 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.761997 15733 catalog_manager.cc:5484] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.762804 17928 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.763685 18190 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:46.766901 16265 consensus_queue.cc:237] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:46.769176 17927 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:46.769302 16134 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.770259 19159 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:46.771044 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:46.771975 17793 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:46.772563 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:46.772835 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet eb90224d08f84a2faf7dcc063c6af56e with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:46.773129 19634 tablet_replica.cc:330] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:46.773386 19634 raft_consensus.cc:2229] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:46.773617 19634 raft_consensus.cc:2258] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:46.774804 19634 ts_tablet_manager.cc:1875] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.778775 19662 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:46.782820 19634 ts_tablet_manager.cc:1888] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:46.783004 19634 log.cc:1196] T eb90224d08f84a2faf7dcc063c6af56e P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb90224d08f84a2faf7dcc063c6af56e
I20240501 20:10:46.784111 19496 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:46.785262 16003 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:46.786475 19670 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:46.787020 18190 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:46.796844 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet eb90224d08f84a2faf7dcc063c6af56e (table rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]) successfully deleted
I20240501 20:10:46.797044 19662 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.802116 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b016fc1d28524a02bd762b74c6b08375 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.803350 17927 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.805583 19697 ts_tablet_manager.cc:1888] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 20:10:46.805739 19697 log.cc:1196] T b016fc1d28524a02bd762b74c6b08375 P 047275bda1074ac5a63325e7e1109c61: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-7/wal/wals/b016fc1d28524a02bd762b74c6b08375
I20240501 20:10:46.806478 15719 catalog_manager.cc:4830] TS 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): tablet b016fc1d28524a02bd762b74c6b08375 (table rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]) successfully deleted
I20240501 20:10:46.808284 17793 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.809732 16005 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.809978 15726 catalog_manager.cc:5484] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 9 to 10, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:46.822710 16265 consensus_queue.cc:237] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:46.831620 17927 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:46.832154 19496 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:46.834421 19159 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.837944 15853 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:46.837963 15854 tablet_service.cc:1514] Processing DeleteTablet for tablet b016fc1d28524a02bd762b74c6b08375 with delete_type TABLET_DATA_TOMBSTONED (TS 51b81a5ae56b4dd99af1b0167c1da1e2 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35708
I20240501 20:10:46.838373 19679 tablet_replica.cc:330] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2: stopping tablet replica
I20240501 20:10:46.838575 19679 raft_consensus.cc:2229] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:46.839068 19679 raft_consensus.cc:2258] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:46.840250 19679 ts_tablet_manager.cc:1875] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.840386 19597 ts_tablet_manager.cc:1888] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:46.840490 19597 log.cc:1196] T e483217b8761457d84edc49253877719 P f856e0d1074b431eb19763847958b89e: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/e483217b8761457d84edc49253877719
I20240501 20:10:46.842661 15718 catalog_manager.cc:4830] TS f856e0d1074b431eb19763847958b89e (127.8.68.196:36919): tablet e483217b8761457d84edc49253877719 (table rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]) successfully deleted
I20240501 20:10:46.844148 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b016fc1d28524a02bd762b74c6b08375 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:46.845229 17927 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.853698 19679 ts_tablet_manager.cc:1888] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:10:46.853936 19679 log.cc:1196] T b016fc1d28524a02bd762b74c6b08375 P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/b016fc1d28524a02bd762b74c6b08375
I20240501 20:10:46.854619 17793 raft_consensus.cc:1261] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:46.854728 15719 catalog_manager.cc:4830] TS 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): tablet b016fc1d28524a02bd762b74c6b08375 (table rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]) successfully deleted
I20240501 20:10:46.855197 19662 consensus_queue.cc:1035] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:46.856914 17793 raft_consensus.cc:2932] T b016fc1d28524a02bd762b74c6b08375 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.863227 15875 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:46.876750 15726 catalog_manager.cc:5484] T b016fc1d28524a02bd762b74c6b08375 P d3917663631a4479b15355381486591c reported cstate change: config changed from index 10 to 11, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:46.885114 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet b016fc1d28524a02bd762b74c6b08375 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:46.885463 19659 tablet_replica.cc:330] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:46.885671 19659 raft_consensus.cc:2229] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:46.886248 19659 raft_consensus.cc:2258] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:46.887312 19659 ts_tablet_manager.cc:1875] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:46.889436 18059 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.889979 16136 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:46.891218 19522 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:10:46.891633 15808 consensus_peers.cc:485] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e6b15e4aebf4b1f9a3176f87083703e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:46.893013 19522 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:46.901886 19659 ts_tablet_manager.cc:1888] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 20:10:46.905014 18190 raft_consensus.cc:1203] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:46.905253 19659 log.cc:1196] T b016fc1d28524a02bd762b74c6b08375 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/b016fc1d28524a02bd762b74c6b08375
I20240501 20:10:46.906045 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet b016fc1d28524a02bd762b74c6b08375 (table rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]) successfully deleted
I20240501 20:10:46.923187 19522 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.924929 18060 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.934090 15725 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.945250 16134 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:46.963840 16005 consensus_queue.cc:237] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:46.967124 19670 raft_consensus.cc:1050] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:46.967958 19670 consensus_queue.cc:237] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:46.984038 15875 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:46.984719 19059 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:46.989020 16003 raft_consensus.cc:1261] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.989428 18059 raft_consensus.cc:1261] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.989835 16136 raft_consensus.cc:1261] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:46.989948 19670 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.990675 19662 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.995671 19670 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:46.996606 16265 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:46.997099 19059 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.030850 19059 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:47.032979 19670 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:47.039412 16263 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:47.043414 19312 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.043349 15725 catalog_manager.cc:5484] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:47.045637 15875 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:47.051237 18059 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:47.056002 19312 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:47.056281 19312 ts_tablet_manager.cc:1367] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.563s	user 0.046s	sys 0.005s
I20240501 20:10:47.061774 19312 raft_consensus.cc:348] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:47.062018 19312 raft_consensus.cc:729] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:47.062224 19312 consensus_queue.cc:260] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:47.063462 19312 ts_tablet_manager.cc:1398] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240501 20:10:47.063251 15726 catalog_manager.cc:5484] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:47.064141 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-ddf5ddac08364043bf2e27184d5d3213 received from {username='slave'} at 127.8.68.200:35897
I20240501 20:10:47.064252 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-ddf5ddac08364043bf2e27184d5d3213 on tablet ddf5ddac08364043bf2e27184d5d3213 with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:47.064074 16005 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:47.078264 16134 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:47.089164 16263 consensus_queue.cc:237] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:47.091687 18059 raft_consensus.cc:1261] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:47.092227 19662 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:47.094591 19662 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:47.099201 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3ba52a871714255aa1bd34d9daca920 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:47.099359 19742 ts_tablet_manager.cc:902] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:47.101040 18059 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:47.102391 15725 catalog_manager.cc:5484] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:47.105988 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ba52a871714255aa1bd34d9daca920 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:47.106331 19659 tablet_replica.cc:330] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:47.106508 19659 raft_consensus.cc:2229] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:47.106696 19659 raft_consensus.cc:2258] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:47.107590 19659 ts_tablet_manager.cc:1875] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:47.110380 16136 raft_consensus.cc:1261] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:47.112376 19662 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:47.114374 19659 ts_tablet_manager.cc:1888] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:47.114081 16134 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:47.114553 19659 log.cc:1196] T d3ba52a871714255aa1bd34d9daca920 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/d3ba52a871714255aa1bd34d9daca920
I20240501 20:10:47.115298 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet d3ba52a871714255aa1bd34d9daca920 (table rebalance_test_table_7 [id=6757c3c851a94b17ae18e261f7bb22d5]) successfully deleted
W20240501 20:10:47.127660 15942 consensus_peers.cc:485] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e0746bed2cf43b9bc7765c270a8aa62. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.150338 19742 tablet_copy_client.cc:308] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:47.152182 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet d48de8e0c12b432ea5079390400302c1 from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:35897)
I20240501 20:10:47.152346 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet d48de8e0c12b432ea5079390400302c1 from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:35897: session id = 047275bda1074ac5a63325e7e1109c61-d48de8e0c12b432ea5079390400302c1
I20240501 20:10:47.155721 15895 tablet_copy_source_session.cc:205] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.158499 16263 consensus_queue.cc:237] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:47.166854 19742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48de8e0c12b432ea5079390400302c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.171948 16005 consensus_queue.cc:237] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:47.177841 18190 raft_consensus.cc:1203] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:47.182044 19742 tablet_copy_client.cc:783] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.182348 19742 tablet_copy_client.cc:647] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.184881 19275 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.185046 19742 tablet_copy_client.cc:523] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.186026 16136 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:47.186667 19662 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:47.190145 19275 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
W20240501 20:10:47.190423 16202 consensus_peers.cc:485] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1474070bbdf147d2ae3313fdf9d71c19. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:47.191519 16202 consensus_peers.cc:485] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.190469 19275 ts_tablet_manager.cc:1367] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.437s	user 0.043s	sys 0.007s
I20240501 20:10:47.197916 19275 raft_consensus.cc:348] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:47.200414 19275 raft_consensus.cc:729] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:47.200699 19275 consensus_queue.cc:260] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:47.201478 15875 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:47.202035 19275 ts_tablet_manager.cc:1398] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 20:10:47.202140 19662 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:47.210347 19742 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:47.211072 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-1474070bbdf147d2ae3313fdf9d71c19 received from {username='slave'} at 127.8.68.199:39585
I20240501 20:10:47.211196 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session cad00548f33849e7a19cecd760e1e429-1474070bbdf147d2ae3313fdf9d71c19 on tablet 1474070bbdf147d2ae3313fdf9d71c19 with peer cad00548f33849e7a19cecd760e1e429
W20240501 20:10:47.212404 15942 consensus_peers.cc:485] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f638f341ff2341dc82889334eb87e3d0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.212755 15874 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.213582 16133 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.214607 19059 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.215188 19059 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.224767 19662 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.233896 15726 catalog_manager.cc:5484] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:47.229584 16136 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.237200 19738 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:47.242228 15874 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:47.246595 15726 catalog_manager.cc:5484] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:47.247789 16133 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:47.259292 16136 consensus_queue.cc:237] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:47.263536 15875 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.276803 16005 raft_consensus.cc:1261] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.277395 18815 consensus_queue.cc:1035] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:47.291999 16072 consensus_peers.cc:485] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff11a8e00b544b4193a2824541668492. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.293108 15875 raft_consensus.cc:1261] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.293679 18815 consensus_queue.cc:1035] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.313817 15874 consensus_queue.cc:237] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
W20240501 20:10:47.327347 15811 consensus_peers.cc:485] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddf5ddac08364043bf2e27184d5d3213. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.328379 18190 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:47.329003 19522 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:47.336086 16263 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:47.336910 19522 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:47.348695 19670 raft_consensus.cc:1050] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
I20240501 20:10:47.349495 19670 consensus_queue.cc:237] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:47.338606 19797 raft_consensus.cc:2932] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.355063 15875 raft_consensus.cc:2932] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.355816 19742 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.356400 19742 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:47.356774 19742 ts_tablet_manager.cc:1367] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.147s	user 0.045s	sys 0.010s
I20240501 20:10:47.357704 19742 raft_consensus.cc:348] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:47.358014 19742 raft_consensus.cc:729] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:47.358273 19742 consensus_queue.cc:260] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:47.358904 19742 ts_tablet_manager.cc:1398] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:47.359855 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-d48de8e0c12b432ea5079390400302c1 received from {username='slave'} at 127.8.68.200:35897
I20240501 20:10:47.360004 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-d48de8e0c12b432ea5079390400302c1 on tablet d48de8e0c12b432ea5079390400302c1 with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:47.362507 15725 catalog_manager.cc:5484] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.366741 19275 ts_tablet_manager.cc:902] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:47.366837 15875 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:47.367702 18189 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:47.367871 19662 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:47.368702 19662 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:47.371356 16005 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:47.371986 19522 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:10:47.379223 16071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:46677, user_credentials={real_user=slave}} blocked reactor thread for 85194us
I20240501 20:10:47.379367 19519 raft_consensus.cc:1013] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: attempt to promote peer 047275bda1074ac5a63325e7e1109c61: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:10:47.381165 18852 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.383684 16136 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:47.384245 19662 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:47.386675 16005 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.386901 16263 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.393898 15874 consensus_queue.cc:237] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:47.393844 15726 catalog_manager.cc:5484] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
W20240501 20:10:47.417764 15811 consensus_peers.cc:485] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3eb13093b546d38fc71368f2083878. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.418767 16005 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.419632 18190 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.422510 19522 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.423249 19664 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:47.427189 15874 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:47.434839 16136 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:47.435784 15728 catalog_manager.cc:5484] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:47.438611 19522 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.442948 16005 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.448351 16263 consensus_queue.cc:237] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:47.445783 18190 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:47.451694 15733 catalog_manager.cc:5484] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.452980 15874 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:47.453444 19662 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:47.460052 19522 raft_consensus.cc:1050] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
I20240501 20:10:47.460958 19522 consensus_queue.cc:237] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
W20240501 20:10:47.463908 15811 consensus_peers.cc:485] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddf5ddac08364043bf2e27184d5d3213. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.466135 18188 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:47.466696 19519 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:47.469759 16263 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:47.470503 19519 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:47.473397 16263 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.475862 18189 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:47.477375 19662 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:47.479380 16005 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:47.480070 19519 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:47.481791 15725 catalog_manager.cc:5484] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.482669 16005 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.487555 18852 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.502068 19662 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:47.507229 19818 ts_tablet_manager.cc:902] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:47.508031 18189 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:47.515516 15874 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:47.516459 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98eb95ded28d433aad9523586e29947f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:47.517217 18189 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.517699 15733 catalog_manager.cc:5484] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 reported cstate change: config changed from index 5 to 6, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:47.522215 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 98eb95ded28d433aad9523586e29947f with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:47.529892 15874 consensus_queue.cc:237] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
W20240501 20:10:47.532668 15811 consensus_peers.cc:485] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddf5ddac08364043bf2e27184d5d3213. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.533150 16265 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:47.537300 18189 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:47.537506 19275 tablet_copy_client.cc:308] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:47.537510 19809 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:47.542042 18852 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.543329 16265 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.552060 19522 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:47.553166 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ddf5ddac08364043bf2e27184d5d3213 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:47.552983 15728 catalog_manager.cc:5484] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 6 to 7, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.557452 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf5ddac08364043bf2e27184d5d3213 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:47.557827 19659 tablet_replica.cc:330] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:47.558053 19659 raft_consensus.cc:2229] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:47.558394 19659 raft_consensus.cc:2258] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:47.559307 19659 ts_tablet_manager.cc:1875] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:47.565758 19659 ts_tablet_manager.cc:1888] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:10:47.565932 19659 log.cc:1196] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/ddf5ddac08364043bf2e27184d5d3213
I20240501 20:10:47.566710 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet ddf5ddac08364043bf2e27184d5d3213 (table rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]) successfully deleted
I20240501 20:10:47.573215 19825 tablet_replica.cc:330] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:47.573500 19825 raft_consensus.cc:2229] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:47.573886 19825 raft_consensus.cc:2258] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:47.574038 18189 raft_consensus.cc:1203] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:47.576995 18190 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.580318 19818 tablet_copy_client.cc:308] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:47.581140 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet 2e6b15e4aebf4b1f9a3176f87083703e from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:60703)
I20240501 20:10:47.581301 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet 2e6b15e4aebf4b1f9a3176f87083703e from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:60703: session id = d3917663631a4479b15355381486591c-2e6b15e4aebf4b1f9a3176f87083703e
I20240501 20:10:47.583916 15895 tablet_copy_source_session.cc:205] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.584095 19825 ts_tablet_manager.cc:1875] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:47.584859 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet 8e0746bed2cf43b9bc7765c270a8aa62 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:37351)
I20240501 20:10:47.585049 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet 8e0746bed2cf43b9bc7765c270a8aa62 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:37351: session id = cad00548f33849e7a19cecd760e1e429-8e0746bed2cf43b9bc7765c270a8aa62
I20240501 20:10:47.587720 16026 tablet_copy_source_session.cc:205] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.587956 19818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6b15e4aebf4b1f9a3176f87083703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.589615 16136 consensus_queue.cc:237] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:47.591280 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0746bed2cf43b9bc7765c270a8aa62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.597465 19275 tablet_copy_client.cc:783] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.600298 19275 tablet_copy_client.cc:647] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.602190 19275 tablet_copy_client.cc:523] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.605528 19275 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:47.609951 19818 tablet_copy_client.cc:783] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.610189 19818 tablet_copy_client.cc:647] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.619781 16005 raft_consensus.cc:2932] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.625650 16133 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.626997 18059 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.627692 15874 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:47.628253 19791 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:47.628687 17793 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:47.628985 19809 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.632416 19791 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:10:47.632769 16072 consensus_peers.cc:485] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36465c82164a4765a63590cfd5d413a4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.638509 19818 tablet_copy_client.cc:523] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.642102 19818 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:47.664393 19815 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:47.674949 15874 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:47.679071 15725 catalog_manager.cc:5484] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:47.729842 19857 ts_tablet_manager.cc:902] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:47.730902 19850 ts_tablet_manager.cc:902] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:47.731628 16136 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.741935 19855 ts_tablet_manager.cc:902] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:47.748435 17793 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:47.754985 19825 ts_tablet_manager.cc:1888] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:47.755178 19825 log.cc:1196] T 98eb95ded28d433aad9523586e29947f P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/98eb95ded28d433aad9523586e29947f
I20240501 20:10:47.756053 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet 98eb95ded28d433aad9523586e29947f (table rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]) successfully deleted
I20240501 20:10:47.780959 19850 tablet_copy_client.cc:308] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:47.786753 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet f638f341ff2341dc82889334eb87e3d0 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:37351)
I20240501 20:10:47.786903 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet f638f341ff2341dc82889334eb87e3d0 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:37351: session id = cad00548f33849e7a19cecd760e1e429-f638f341ff2341dc82889334eb87e3d0
I20240501 20:10:47.789783 16026 tablet_copy_source_session.cc:205] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.791249 19850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f638f341ff2341dc82889334eb87e3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.800709 19857 tablet_copy_client.cc:308] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:47.802847 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet 1474070bbdf147d2ae3313fdf9d71c19 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:55615)
I20240501 20:10:47.802995 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet 1474070bbdf147d2ae3313fdf9d71c19 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:55615: session id = 9a0f0037700246bdb65e05aef95fb2be-1474070bbdf147d2ae3313fdf9d71c19
I20240501 20:10:47.805784 16285 tablet_copy_source_session.cc:205] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.807150 19857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1474070bbdf147d2ae3313fdf9d71c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.810612 19855 tablet_copy_client.cc:308] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:47.812098 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet ff11a8e00b544b4193a2824541668492 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:53423)
I20240501 20:10:47.812260 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet ff11a8e00b544b4193a2824541668492 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:53423: session id = 9a0f0037700246bdb65e05aef95fb2be-ff11a8e00b544b4193a2824541668492
I20240501 20:10:47.815493 16156 tablet_copy_source_session.cc:205] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.822948 16265 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:47.823750 19275 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.824151 19275 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:47.824388 19275 ts_tablet_manager.cc:1367] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.219s	user 0.053s	sys 0.023s
I20240501 20:10:47.827189 19892 ts_tablet_manager.cc:902] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:47.827308 19275 raft_consensus.cc:348] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:47.827575 19275 raft_consensus.cc:729] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:47.827776 19275 consensus_queue.cc:260] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:47.834398 19857 tablet_copy_client.cc:783] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.834841 19857 tablet_copy_client.cc:647] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
W20240501 20:10:47.836477 16199 consensus_peers.cc:485] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c538581c71f646b5bc8d71cb6981f201. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.836517 16136 consensus_queue.cc:237] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:47.836951 19275 ts_tablet_manager.cc:1398] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.000s
I20240501 20:10:47.839648 16134 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:47.839998 19857 tablet_copy_client.cc:523] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.840163 16026 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-8e0746bed2cf43b9bc7765c270a8aa62 received from {username='slave'} at 127.8.68.199:37351
I20240501 20:10:47.840338 16026 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session cad00548f33849e7a19cecd760e1e429-8e0746bed2cf43b9bc7765c270a8aa62 on tablet 8e0746bed2cf43b9bc7765c270a8aa62 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:47.840428 19662 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:47.844036 18060 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:47.844113 19662 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:47.844671 19664 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:47.846026 16134 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:47.847447 19857 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:47.856034 15728 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:47.870604 18060 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:47.872153 19892 tablet_copy_client.cc:308] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:47.874439 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet 8f3eb13093b546d38fc71368f2083878 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:49547)
I20240501 20:10:47.874616 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet 8f3eb13093b546d38fc71368f2083878 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:49547: session id = 9a0f0037700246bdb65e05aef95fb2be-8f3eb13093b546d38fc71368f2083878
I20240501 20:10:47.877368 15895 tablet_copy_source_session.cc:205] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:47.882195 19892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3eb13093b546d38fc71368f2083878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.878832 19818 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:47.883548 19818 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:47.883790 19818 ts_tablet_manager.cc:1367] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.242s	user 0.050s	sys 0.006s
I20240501 20:10:47.884394 19818 raft_consensus.cc:348] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:47.884618 19818 raft_consensus.cc:729] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:47.884802 19818 consensus_queue.cc:260] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:47.889822 19818 ts_tablet_manager.cc:1398] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240501 20:10:47.890472 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session d3917663631a4479b15355381486591c-2e6b15e4aebf4b1f9a3176f87083703e received from {username='slave'} at 127.8.68.198:60703
I20240501 20:10:47.890583 16005 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.890666 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session d3917663631a4479b15355381486591c-2e6b15e4aebf4b1f9a3176f87083703e on tablet 2e6b15e4aebf4b1f9a3176f87083703e with peer d3917663631a4479b15355381486591c
I20240501 20:10:47.891108 19791 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.895450 19855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff11a8e00b544b4193a2824541668492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:47.895133 16005 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:47.894078 19791 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:47.903977 15725 catalog_manager.cc:5484] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:47.922830 19892 tablet_copy_client.cc:783] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.923595 19892 tablet_copy_client.cc:647] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.924492 16265 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:47.924389 19850 tablet_copy_client.cc:783] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.925143 19791 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:47.926705 19850 tablet_copy_client.cc:647] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.927088 16265 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:47.927639 19892 tablet_copy_client.cc:523] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:47.932803 19850 tablet_copy_client.cc:523] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 20:10:47.934259 16069 consensus_peers.cc:485] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f345303c72774848bb60e806cff22d12. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.947661 19850 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:47.948768 19892 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:47.956439 16005 consensus_queue.cc:237] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
W20240501 20:10:47.959405 15942 consensus_peers.cc:485] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f904b62401ff4528b3fa65dd4e11512a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.960716 16263 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:47.961333 19738 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:47.961445 18190 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:47.962214 19059 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:47.963618 19855 tablet_copy_client.cc:783] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:47.965284 19855 tablet_copy_client.cc:647] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:47.975087 16265 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:47.978782 18060 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:47.979463 16136 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:10:47.981128 16202 consensus_peers.cc:485] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3643d6995910404fb8a7e6ef9c635f42. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:47.981477 19670 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:47.983587 19670 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:47.985847 19670 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.989406 18059 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:47.995456 17927 raft_consensus.cc:1203] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 20:10:47.996122 15726 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:48.003468 19857 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:48.003858 19857 tablet_bootstrap.cc:492] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:48.004134 19857 ts_tablet_manager.cc:1367] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.157s	user 0.047s	sys 0.000s
I20240501 20:10:48.004877 19857 raft_consensus.cc:348] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:48.005162 19857 raft_consensus.cc:729] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:48.005337 19857 consensus_queue.cc:260] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:48.005859 19857 ts_tablet_manager.cc:1398] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:10:48.007329 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-1474070bbdf147d2ae3313fdf9d71c19 received from {username='slave'} at 127.8.68.197:55615
I20240501 20:10:48.007485 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-1474070bbdf147d2ae3313fdf9d71c19 on tablet 1474070bbdf147d2ae3313fdf9d71c19 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:48.013953 19855 tablet_copy_client.cc:523] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:48.031994 16134 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:48.034586 19857 ts_tablet_manager.cc:902] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:48.041916 19855 tablet_bootstrap.cc:492] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:48.043218 19275 ts_tablet_manager.cc:902] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:48.059527 16005 consensus_queue.cc:237] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:48.064204 16265 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:10:48.064425 15940 consensus_peers.cc:485] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e0746bed2cf43b9bc7765c270a8aa62. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:48.065070 15873 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:48.065829 19738 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:48.066589 19738 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:48.068884 18059 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:48.070371 19738 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:48.070756 19031 raft_consensus.cc:1050] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:48.071650 19031 consensus_queue.cc:237] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:48.073340 19738 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.074613 16136 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:48.074775 17793 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:48.075132 19662 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:48.075507 19664 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:48.076395 18058 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:48.077555 15873 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.078307 19664 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:48.080080 19275 tablet_copy_client.cc:308] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:48.080793 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet 36465c82164a4765a63590cfd5d413a4 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:53283)
I20240501 20:10:48.080950 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet 36465c82164a4765a63590cfd5d413a4 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:53283: session id = cad00548f33849e7a19cecd760e1e429-36465c82164a4765a63590cfd5d413a4
I20240501 20:10:48.080792 19664 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:48.083825 16156 tablet_copy_source_session.cc:205] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:48.084586 18058 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:48.085161 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36465c82164a4765a63590cfd5d413a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:48.085635 19857 tablet_copy_client.cc:308] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:48.086237 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet ddf5ddac08364043bf2e27184d5d3213 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:49547)
I20240501 20:10:48.086422 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet ddf5ddac08364043bf2e27184d5d3213 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:49547: session id = 9a0f0037700246bdb65e05aef95fb2be-ddf5ddac08364043bf2e27184d5d3213
I20240501 20:10:48.089125 15895 tablet_copy_source_session.cc:205] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:48.089813 19059 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:48.090622 19857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf5ddac08364043bf2e27184d5d3213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:48.093640 16266 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:48.097802 19738 raft_consensus.cc:1050] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:48.105008 15725 catalog_manager.cc:5484] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.108928 19857 tablet_copy_client.cc:783] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:48.109153 19857 tablet_copy_client.cc:647] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:48.109422 18060 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.111459 19857 tablet_copy_client.cc:523] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:48.114697 19857 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:48.116056 19738 consensus_queue.cc:237] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:48.115867 18190 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:48.116513 17793 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:48.126693 18059 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:48.127203 15873 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:48.127830 19891 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:48.128587 19891 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:48.129722 16136 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
W20240501 20:10:48.131968 15940 consensus_peers.cc:485] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e0746bed2cf43b9bc7765c270a8aa62. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:48.132820 15725 catalog_manager.cc:5484] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:48.134493 15728 catalog_manager.cc:5484] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:48.135803 19662 raft_consensus.cc:1050] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:48.136837 19662 consensus_queue.cc:237] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:48.139477 16136 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:48.140023 19664 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:48.139045 19968 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.142383 15873 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:48.144258 19275 tablet_copy_client.cc:783] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:48.146723 15873 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:48.147209 19522 raft_consensus.cc:1050] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
I20240501 20:10:48.147341 19664 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:48.148039 19522 consensus_queue.cc:237] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:48.149458 15873 raft_consensus.cc:858] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
W20240501 20:10:48.149677 15873 raft_consensus.cc:1565] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:48.151623 18059 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:48.152191 19670 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:48.156102 19662 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.160478 18059 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.165344 16136 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.170698 18189 raft_consensus.cc:1261] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:48.171371 19809 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:48.171700 15726 catalog_manager.cc:5484] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.173337 15733 catalog_manager.cc:5484] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.173465 15874 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.150749 15873 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:48.178534 18059 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.179491 15875 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.144488 19275 tablet_copy_client.cc:647] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:48.186682 19275 tablet_copy_client.cc:523] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:48.190021 19275 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:48.198275 16266 raft_consensus.cc:1261] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:48.198858 18852 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:48.201247 17795 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:48.202478 19960 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:48.209260 16005 raft_consensus.cc:1261] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:48.209981 18852 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:48.210036 17793 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.212644 18852 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:48.216168 16263 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:48.217934 19738 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:48.218322 16005 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:48.219239 16263 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.223625 18190 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:48.223975 16263 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.228909 16263 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
W20240501 20:10:48.233465 19975 log.cc:924] Time spent T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20240501 20:10:48.246529 19818 ts_tablet_manager.cc:902] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:48.253804 15734 catalog_manager.cc:5484] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:48.263183 15873 consensus_queue.cc:237] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:48.265404 16265 raft_consensus.cc:1261] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:48.265666 18189 raft_consensus.cc:1261] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:48.265976 19809 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:48.266273 18852 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:48.270067 19522 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:48.272727 18189 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:48.281627 15734 catalog_manager.cc:5484] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 reported cstate change: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:48.282202 16136 consensus_queue.cc:237] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:48.286800 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet d48de8e0c12b432ea5079390400302c1 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:48.298249 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d48de8e0c12b432ea5079390400302c1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:48.299093 16265 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:48.300252 20000 tablet_replica.cc:330] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:48.300742 20000 raft_consensus.cc:2229] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:48.301012 19818 tablet_copy_client.cc:308] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:48.302584 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet c538581c71f646b5bc8d71cb6981f201 from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:46329)
I20240501 20:10:48.302775 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet c538581c71f646b5bc8d71cb6981f201 from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:46329: session id = d3917663631a4479b15355381486591c-c538581c71f646b5bc8d71cb6981f201
I20240501 20:10:48.305631 16285 tablet_copy_source_session.cc:205] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:48.306026 16263 consensus_queue.cc:237] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:48.307364 19818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c538581c71f646b5bc8d71cb6981f201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:48.301405 20000 raft_consensus.cc:2258] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:48.312376 16005 raft_consensus.cc:1261] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:48.314515 15873 raft_consensus.cc:1261] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:48.315244 19797 consensus_queue.cc:1035] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:48.317203 19797 consensus_queue.cc:1035] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:48.321671 19818 tablet_copy_client.cc:783] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:48.322106 19818 tablet_copy_client.cc:647] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:48.325309 19818 tablet_copy_client.cc:523] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:48.335518 19818 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:48.338840 20000 ts_tablet_manager.cc:1875] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:48.347867 19797 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.348711 20000 ts_tablet_manager.cc:1888] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:48.349642 20000 log.cc:1196] T d48de8e0c12b432ea5079390400302c1 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/d48de8e0c12b432ea5079390400302c1
I20240501 20:10:48.351032 15875 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.356889 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet d48de8e0c12b432ea5079390400302c1 (table rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]) successfully deleted
I20240501 20:10:48.365221 15733 catalog_manager.cc:5484] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:48.367825 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:48.375068 16005 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:48.375584 17793 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:48.376277 19960 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:48.379165 18059 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:48.380556 19662 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:48.381630 16136 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:48.382304 19670 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:48.383157 19670 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.385316 16136 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.387167 18059 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.392341 15734 catalog_manager.cc:5484] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 7 to 8, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.396905 15854 tablet_service.cc:1514] Processing DeleteTablet for tablet 1474070bbdf147d2ae3313fdf9d71c19 with delete_type TABLET_DATA_TOMBSTONED (TS 51b81a5ae56b4dd99af1b0167c1da1e2 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35708
I20240501 20:10:48.399394 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1474070bbdf147d2ae3313fdf9d71c19 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:48.400444 17793 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.408983 20013 tablet_replica.cc:330] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2: stopping tablet replica
I20240501 20:10:48.409495 20013 raft_consensus.cc:2229] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:48.410081 20013 raft_consensus.cc:2258] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:48.413663 16263 consensus_queue.cc:237] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:48.416630 17795 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:48.417086 20013 ts_tablet_manager.cc:1875] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:48.417335 19960 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:48.419538 18059 raft_consensus.cc:1261] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:48.419843 19031 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.420455 17795 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.423141 19662 consensus_queue.cc:1035] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:48.424455 18058 raft_consensus.cc:2932] T 1474070bbdf147d2ae3313fdf9d71c19 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:48.430487 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1474070bbdf147d2ae3313fdf9d71c19 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:48.432531 15733 catalog_manager.cc:5484] T 1474070bbdf147d2ae3313fdf9d71c19 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:48.436447 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 1474070bbdf147d2ae3313fdf9d71c19 with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:48.441136 16003 consensus_queue.cc:237] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:48.448966 19857 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:48.449412 19857 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:48.449666 19857 ts_tablet_manager.cc:1367] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.335s	user 0.040s	sys 0.010s
I20240501 20:10:48.450944 19857 raft_consensus.cc:348] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:48.451459 19857 raft_consensus.cc:729] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:48.451675 19857 consensus_queue.cc:260] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:48.452307 19857 ts_tablet_manager.cc:1398] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:10:48.452955 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-ddf5ddac08364043bf2e27184d5d3213 received from {username='slave'} at 127.8.68.197:49547
I20240501 20:10:48.453089 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-ddf5ddac08364043bf2e27184d5d3213 on tablet ddf5ddac08364043bf2e27184d5d3213 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:48.454495 19857 ts_tablet_manager.cc:902] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
W20240501 20:10:48.465662 16070 consensus_peers.cc:485] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84d7bbca7af34a2db637b55635fa2f7a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:48.498642 19275 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:48.499593 19275 tablet_bootstrap.cc:492] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:48.499867 19275 ts_tablet_manager.cc:1367] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.310s	user 0.062s	sys 0.003s
I20240501 20:10:48.501374 19275 raft_consensus.cc:348] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:48.501636 19275 raft_consensus.cc:729] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:48.501915 19275 consensus_queue.cc:260] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:48.504341 19275 ts_tablet_manager.cc:1398] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:10:48.505292 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-36465c82164a4765a63590cfd5d413a4 received from {username='slave'} at 127.8.68.199:53283
I20240501 20:10:48.505434 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session cad00548f33849e7a19cecd760e1e429-36465c82164a4765a63590cfd5d413a4 on tablet 36465c82164a4765a63590cfd5d413a4 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:48.509367 20028 ts_tablet_manager.cc:902] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:48.518661 20020 tablet_replica.cc:330] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:48.518915 20020 raft_consensus.cc:2229] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:48.519377 20020 raft_consensus.cc:2258] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:48.520762 20020 ts_tablet_manager.cc:1875] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:48.530449 20013 ts_tablet_manager.cc:1888] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 20:10:48.530606 20013 log.cc:1196] T 1474070bbdf147d2ae3313fdf9d71c19 P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/1474070bbdf147d2ae3313fdf9d71c19
I20240501 20:10:48.543300 15719 catalog_manager.cc:4830] TS 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): tablet 1474070bbdf147d2ae3313fdf9d71c19 (table rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]) successfully deleted
I20240501 20:10:48.557431 15873 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:48.558104 19738 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:48.560918 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:48.572970 19850 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:48.573539 19850 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:48.573870 19850 ts_tablet_manager.cc:1367] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.626s	user 0.043s	sys 0.008s
I20240501 20:10:48.580380 19738 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:48.584592 18852 raft_consensus.cc:1050] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:48.585413 18852 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:48.591250 15728 catalog_manager.cc:5484] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:48.600659 15873 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:48.619719 17793 raft_consensus.cc:1203] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 20:10:48.624837 16136 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:48.626425 19275 ts_tablet_manager.cc:902] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:48.629165 16005 consensus_queue.cc:237] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:48.630213 20050 ts_tablet_manager.cc:902] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:48.635003 16265 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:48.636447 19059 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:48.659337 18059 raft_consensus.cc:1203] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 20:10:48.661353 15873 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:48.662024 19738 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:48.665481 19891 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:48.671195 15875 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:48.673238 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:48.677572 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8e0746bed2cf43b9bc7765c270a8aa62 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:48.678629 15728 catalog_manager.cc:5484] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:48.683063 18170 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e0746bed2cf43b9bc7765c270a8aa62 with delete_type TABLET_DATA_TOMBSTONED (TS 047275bda1074ac5a63325e7e1109c61 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35670
I20240501 20:10:48.691378 18059 raft_consensus.cc:1261] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:48.574707 19850 raft_consensus.cc:348] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:48.691696 19850 raft_consensus.cc:729] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:48.691920 19850 consensus_queue.cc:260] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:48.692590 19850 ts_tablet_manager.cc:1398] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.119s	user 0.002s	sys 0.000s
W20240501 20:10:48.693301 15942 consensus_peers.cc:485] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:48.694392 19738 consensus_queue.cc:1035] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240501 20:10:48.697177 19891 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:48.723922 20020 ts_tablet_manager.cc:1888] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:48.725690 20020 log.cc:1196] T 1474070bbdf147d2ae3313fdf9d71c19 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/1474070bbdf147d2ae3313fdf9d71c19
W20240501 20:10:48.736459 15719 catalog_manager.cc:4797] TS 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): delete failed for tablet 8e0746bed2cf43b9bc7765c270a8aa62 because the tablet was not found. No further retry: Not found: Tablet not found: 8e0746bed2cf43b9bc7765c270a8aa62
I20240501 20:10:48.737463 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet 1474070bbdf147d2ae3313fdf9d71c19 (table rebalance_test_table_4 [id=c26f53ebc0e84a86904dd9a39a23b960]) successfully deleted
I20240501 20:10:48.742120 20068 ts_tablet_manager.cc:902] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:48.741818 16263 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:48.751911 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:48.752234 16134 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:48.761633 19809 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20240501 20:10:48.814532 16136 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
W20240501 20:10:48.820219 15942 consensus_peers.cc:485] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f638f341ff2341dc82889334eb87e3d0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:48.699333 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:48.748155 18059 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:48.818403 18058 raft_consensus.cc:2932] T 8e0746bed2cf43b9bc7765c270a8aa62 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:48.826517 19809 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:48.866178 16026 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-f638f341ff2341dc82889334eb87e3d0 received from {username='slave'} at 127.8.68.199:37351
I20240501 20:10:48.866371 16026 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session cad00548f33849e7a19cecd760e1e429-f638f341ff2341dc82889334eb87e3d0 on tablet f638f341ff2341dc82889334eb87e3d0 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:48.874709 20028 tablet_copy_client.cc:308] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:48.876895 17929 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:48.878089 19809 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:48.890012 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:48.893896 20050 tablet_copy_client.cc:308] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:48.894649 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet 8e0746bed2cf43b9bc7765c270a8aa62 from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:49021)
I20240501 20:10:48.894871 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet 8e0746bed2cf43b9bc7765c270a8aa62 from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:49021: session id = 047275bda1074ac5a63325e7e1109c61-8e0746bed2cf43b9bc7765c270a8aa62
I20240501 20:10:48.897898 16026 tablet_copy_source_session.cc:205] T 8e0746bed2cf43b9bc7765c270a8aa62 P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:48.899432 20050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0746bed2cf43b9bc7765c270a8aa62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:48.905735 20050 tablet_copy_client.cc:783] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:48.906049 20050 tablet_copy_client.cc:647] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:48.908133 20050 tablet_copy_client.cc:523] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:48.911603 20050 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:48.933542 19818 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:48.940495 18852 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:48.947256 16132 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:48.948683 17929 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:48.955394 15728 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:48.959059 19275 tablet_copy_client.cc:308] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:48.962448 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet f904b62401ff4528b3fa65dd4e11512a from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:37351)
I20240501 20:10:48.962597 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet f904b62401ff4528b3fa65dd4e11512a from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:37351: session id = cad00548f33849e7a19cecd760e1e429-f904b62401ff4528b3fa65dd4e11512a
I20240501 20:10:48.963310 19818 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:48.963649 19818 ts_tablet_manager.cc:1367] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.628s	user 0.052s	sys 0.006s
I20240501 20:10:48.964464 19818 raft_consensus.cc:348] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:48.964664 19818 raft_consensus.cc:729] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:48.964866 19818 consensus_queue.cc:260] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:48.965484 16026 tablet_copy_source_session.cc:205] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:48.965431 18060 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:48.968761 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f904b62401ff4528b3fa65dd4e11512a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:48.971961 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet f345303c72774848bb60e806cff22d12 from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:48071)
I20240501 20:10:48.972172 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet f345303c72774848bb60e806cff22d12 from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:48071: session id = d3917663631a4479b15355381486591c-f345303c72774848bb60e806cff22d12
I20240501 20:10:48.975490 16156 tablet_copy_source_session.cc:205] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:48.976784 20028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f345303c72774848bb60e806cff22d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:48.977185 19892 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:48.977571 19892 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:48.977797 19892 ts_tablet_manager.cc:1367] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 1.029s	user 0.039s	sys 0.007s
I20240501 20:10:48.978528 19892 raft_consensus.cc:348] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:48.978778 19892 raft_consensus.cc:729] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:48.978960 19892 consensus_queue.cc:260] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:48.987258 20105 ts_tablet_manager.cc:902] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677)
I20240501 20:10:48.993511 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:49.016937 20050 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:49.017372 20050 tablet_bootstrap.cc:492] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:49.017648 20050 ts_tablet_manager.cc:1367] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.106s	user 0.039s	sys 0.010s
I20240501 20:10:49.020478 19818 ts_tablet_manager.cc:1398] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.057s	user 0.002s	sys 0.000s
I20240501 20:10:49.021848 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session d3917663631a4479b15355381486591c-c538581c71f646b5bc8d71cb6981f201 received from {username='slave'} at 127.8.68.198:46329
I20240501 20:10:49.021960 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session d3917663631a4479b15355381486591c-c538581c71f646b5bc8d71cb6981f201 on tablet c538581c71f646b5bc8d71cb6981f201 with peer d3917663631a4479b15355381486591c
I20240501 20:10:49.026902 20028 tablet_copy_client.cc:783] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:49.027201 20028 tablet_copy_client.cc:647] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:49.029266 20028 tablet_copy_client.cc:523] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:49.033008 20028 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:49.038923 20050 raft_consensus.cc:348] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:49.039193 20050 raft_consensus.cc:729] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: NON_PARTICIPANT
I20240501 20:10:49.039393 20050 consensus_queue.cc:260] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:49.041026 19892 ts_tablet_manager.cc:1398] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.063s	user 0.003s	sys 0.000s
I20240501 20:10:49.044461 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-8f3eb13093b546d38fc71368f2083878 received from {username='slave'} at 127.8.68.197:49547
I20240501 20:10:49.044598 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-8f3eb13093b546d38fc71368f2083878 on tablet 8f3eb13093b546d38fc71368f2083878 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:49.044652 19857 tablet_copy_client.cc:308] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:49.045557 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet 3643d6995910404fb8a7e6ef9c635f42 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:55615)
I20240501 20:10:49.045691 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet 3643d6995910404fb8a7e6ef9c635f42 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:55615: session id = 9a0f0037700246bdb65e05aef95fb2be-3643d6995910404fb8a7e6ef9c635f42
I20240501 20:10:49.051155 20105 tablet_copy_client.cc:308] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:46677
I20240501 20:10:49.070075 16285 tablet_copy_source_session.cc:205] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:49.072443 19857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3643d6995910404fb8a7e6ef9c635f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.078804 19857 tablet_copy_client.cc:783] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:49.079018 19857 tablet_copy_client.cc:647] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:49.081692 19857 tablet_copy_client.cc:523] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:49.086935 19855 tablet_bootstrap.cc:492] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:49.093740 19855 tablet_bootstrap.cc:492] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:49.096163 19855 ts_tablet_manager.cc:1367] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 1.054s	user 0.053s	sys 0.010s
I20240501 20:10:49.097025 19855 raft_consensus.cc:348] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:49.099480 19855 raft_consensus.cc:729] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:49.099752 19855 consensus_queue.cc:260] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:49.099380 19857 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:49.107599 16026 tablet_copy_service.cc:140] P 754fcd3b82f041568f546c6205f44272: Received BeginTabletCopySession request for tablet f638f341ff2341dc82889334eb87e3d0 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:39757)
I20240501 20:10:49.107769 16026 tablet_copy_service.cc:161] P 754fcd3b82f041568f546c6205f44272: Beginning new tablet copy session on tablet f638f341ff2341dc82889334eb87e3d0 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:39757: session id = 9a0f0037700246bdb65e05aef95fb2be-f638f341ff2341dc82889334eb87e3d0
I20240501 20:10:49.109184 19855 ts_tablet_manager.cc:1398] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.000s
I20240501 20:10:49.110664 16026 tablet_copy_source_session.cc:205] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:49.112018 20105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f638f341ff2341dc82889334eb87e3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.118563 19797 raft_consensus.cc:1050] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:49.119340 20105 tablet_copy_client.cc:783] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:49.119750 20105 tablet_copy_client.cc:647] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:49.121071 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-ff11a8e00b544b4193a2824541668492 received from {username='slave'} at 127.8.68.197:53423
I20240501 20:10:49.121183 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-ff11a8e00b544b4193a2824541668492 on tablet ff11a8e00b544b4193a2824541668492 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:49.128990 19797 consensus_queue.cc:237] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:49.131665 20050 ts_tablet_manager.cc:1398] T 8e0746bed2cf43b9bc7765c270a8aa62 P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.114s	user 0.013s	sys 0.008s
I20240501 20:10:49.131875 20105 tablet_copy_client.cc:523] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:49.133301 16026 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-8e0746bed2cf43b9bc7765c270a8aa62 received from {username='slave'} at 127.8.68.200:49021
I20240501 20:10:49.133404 16026 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-8e0746bed2cf43b9bc7765c270a8aa62 on tablet 8e0746bed2cf43b9bc7765c270a8aa62 with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:49.134918 20105 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:49.151232 18059 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:49.157477 18058 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.158077 19808 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.165647 17793 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.166270 19808 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.170780 19809 raft_consensus.cc:1050] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:49.171648 19809 consensus_queue.cc:237] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:49.175271 17927 raft_consensus.cc:1203] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 20:10:49.194101 19808 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:49.199973 17793 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:49.203526 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:49.206759 16263 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:49.208801 15733 catalog_manager.cc:5484] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be reported cstate change: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:49.210388 18059 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:49.217739 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:10:49.222239 16199 consensus_peers.cc:485] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3643d6995910404fb8a7e6ef9c635f42. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:49.223827 16132 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.226240 15873 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.230013 19960 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.235383 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:49.237130 16202 consensus_peers.cc:485] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:49.239284 18059 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.241437 19960 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.252854 19031 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:49.253875 19275 tablet_copy_client.cc:783] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:49.254122 19275 tablet_copy_client.cc:647] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:49.258802 16136 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:49.259305 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:49.268361 15728 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 7 to 8, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:49.279814 20109 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.281244 20068 tablet_copy_client.cc:308] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:49.281601 16136 consensus_queue.cc:237] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:49.282560 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:49.283954 18059 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:49.284547 19797 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:49.287065 19797 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:49.288336 19275 tablet_copy_client.cc:523] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:49.293709 18059 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:49.298661 18190 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.299309 20073 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.310817 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:49.311285 16263 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.311812 18852 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.311689 15733 catalog_manager.cc:5484] T 36465c82164a4765a63590cfd5d413a4 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 8 to 9, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:49.316505 15854 tablet_service.cc:1514] Processing DeleteTablet for tablet 36465c82164a4765a63590cfd5d413a4 with delete_type TABLET_DATA_TOMBSTONED (TS 51b81a5ae56b4dd99af1b0167c1da1e2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35708
I20240501 20:10:49.329751 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:49.338421 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 36465c82164a4765a63590cfd5d413a4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:49.350749 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet 84d7bbca7af34a2db637b55635fa2f7a from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:48881)
I20240501 20:10:49.350919 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet 84d7bbca7af34a2db637b55635fa2f7a from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:48881: session id = 047275bda1074ac5a63325e7e1109c61-84d7bbca7af34a2db637b55635fa2f7a
I20240501 20:10:49.353932 16156 tablet_copy_source_session.cc:205] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:49.357345 17794 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.358045 17793 raft_consensus.cc:1261] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:49.358753 20109 consensus_queue.cc:1035] T 36465c82164a4765a63590cfd5d413a4 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:49.358796 18059 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:49.360563 18852 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.360378 17794 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:49.373793 19522 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:49.377373 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:49.378877 18190 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:49.380731 15733 catalog_manager.cc:5484] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:49.383052 16263 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:49.393035 20068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d7bbca7af34a2db637b55635fa2f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.400705 19275 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:49.402251 15875 raft_consensus.cc:2932] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:49.406117 17795 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:49.407675 20165 tablet_replica.cc:330] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2: stopping tablet replica
I20240501 20:10:49.407871 20165 raft_consensus.cc:2229] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:49.408100 20165 raft_consensus.cc:2258] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:49.489809 20165 ts_tablet_manager.cc:1875] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:49.494060 15875 consensus_queue.cc:237] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:49.496171 17795 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:49.496762 18190 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:49.497509 19809 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:49.498540 20165 ts_tablet_manager.cc:1888] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:49.498863 20165 log.cc:1196] T 36465c82164a4765a63590cfd5d413a4 P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/36465c82164a4765a63590cfd5d413a4
I20240501 20:10:49.499747 19809 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:49.500636 15719 catalog_manager.cc:4830] TS 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): tablet 36465c82164a4765a63590cfd5d413a4 (table rebalance_test_table_8 [id=322459d71dfc45c7b076dfc0b0076901]) successfully deleted
I20240501 20:10:49.503029 20073 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:49.509244 17793 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:49.514750 17794 raft_consensus.cc:1203] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:49.515287 17792 raft_consensus.cc:1203] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:49.516927 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ddf5ddac08364043bf2e27184d5d3213 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:49.517021 18190 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:49.520576 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:49.526569 16136 consensus_queue.cc:237] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:49.537168 15734 catalog_manager.cc:5484] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:10:49.543316 16070 consensus_peers.cc:485] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5682259bc54c469e9c6a08723c1bbee0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:49.544281 17793 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:49.544517 16265 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:49.545358 19797 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:49.545933 19797 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:49.548871 20109 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:49.553072 16265 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:49.554325 15727 catalog_manager.cc:5484] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:49.569738 17793 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:49.585247 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf5ddac08364043bf2e27184d5d3213 with delete_type TABLET_DATA_TOMBSTONED (TS f856e0d1074b431eb19763847958b89e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44866
I20240501 20:10:49.608597 20068 tablet_copy_client.cc:783] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:49.611531 20068 tablet_copy_client.cc:647] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:49.618840 20068 tablet_copy_client.cc:523] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:49.627295 20189 tablet_replica.cc:330] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e: stopping tablet replica
I20240501 20:10:49.627985 20189 raft_consensus.cc:2229] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:49.628237 20189 raft_consensus.cc:2258] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:49.645092 20068 tablet_bootstrap.cc:492] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:49.652209 20189 ts_tablet_manager.cc:1875] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:49.684715 15875 consensus_queue.cc:237] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:49.714056 17793 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:49.714745 16003 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:49.714730 19522 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:49.716925 16136 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:10:49.719158 15811 consensus_peers.cc:485] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3eb13093b546d38fc71368f2083878. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:49.719286 20073 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:49.719715 20073 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:49.722271 20159 raft_consensus.cc:1050] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:49.725852 20159 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:49.731969 19738 raft_consensus.cc:1050] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:49.732811 19738 consensus_queue.cc:237] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:49.734380 19809 raft_consensus.cc:1013] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: attempt to promote peer 9a0f0037700246bdb65e05aef95fb2be: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:10:49.781343 20208 ts_tablet_manager.cc:902] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:49.815235 20189 ts_tablet_manager.cc:1888] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:49.815461 20189 log.cc:1196] T ddf5ddac08364043bf2e27184d5d3213 P f856e0d1074b431eb19763847958b89e: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/ddf5ddac08364043bf2e27184d5d3213
I20240501 20:10:49.816591 15718 catalog_manager.cc:4830] TS f856e0d1074b431eb19763847958b89e (127.8.68.196:36919): tablet ddf5ddac08364043bf2e27184d5d3213 (table rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]) successfully deleted
I20240501 20:10:49.819180 20208 tablet_copy_client.cc:308] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:49.819947 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet 3643d6995910404fb8a7e6ef9c635f42 from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:46329)
I20240501 20:10:49.820132 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet 3643d6995910404fb8a7e6ef9c635f42 from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:46329: session id = d3917663631a4479b15355381486591c-3643d6995910404fb8a7e6ef9c635f42
I20240501 20:10:49.821389 16132 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:49.822190 19059 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:49.823364 16285 tablet_copy_source_session.cc:205] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:49.824656 20208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3643d6995910404fb8a7e6ef9c635f42. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:10:49.826098 15942 consensus_peers.cc:485] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:49.828863 18058 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:49.829460 19059 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:49.831208 20208 tablet_copy_client.cc:783] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:49.831418 20208 tablet_copy_client.cc:647] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:49.831816 19059 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:49.832834 20208 tablet_copy_client.cc:523] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:49.833305 15875 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:49.833896 19891 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:49.834791 18058 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:49.835953 20208 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:49.841161 19522 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:49.844892 16132 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:49.852571 15733 catalog_manager.cc:5484] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:49.861410 17793 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:49.870913 18852 raft_consensus.cc:1050] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:49.871750 18852 consensus_queue.cc:237] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
W20240501 20:10:49.875572 15811 consensus_peers.cc:485] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3eb13093b546d38fc71368f2083878. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:49.876377 17792 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:49.877683 20073 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:49.884785 15734 catalog_manager.cc:5484] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:49.890245 16132 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:49.890863 19522 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:49.894073 17929 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.894770 19670 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.899718 18058 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.900203 19031 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.903630 16134 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:49.904525 19670 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:49.907509 19031 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:49.911047 18058 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:49.920936 15734 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:49.938349 17929 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:49.951102 16263 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:49.962004 18058 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:49.962707 19031 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:10:49.976035 20205 log.cc:924] Time spent T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20240501 20:10:49.976562 16004 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:49.977371 19522 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:49.978655 16004 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:49.984776 16136 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:49.990970 20159 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:49.992689 19522 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:49.995028 16135 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:49.998189 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c538581c71f646b5bc8d71cb6981f201 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:49.998055 17793 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:49.999662 18058 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:50.007393 15733 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:50.012220 15728 catalog_manager.cc:5484] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 5 to 6, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:50.021543 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet c538581c71f646b5bc8d71cb6981f201 with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:50.028389 15873 consensus_queue.cc:237] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:50.031174 16136 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:50.031915 20073 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:50.035006 17793 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:50.035722 20073 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:50.040750 20237 ts_tablet_manager.cc:902] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:50.047102 16003 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:50.047708 19522 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:50.048785 16004 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:50.069346 20073 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.075001 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f3eb13093b546d38fc71368f2083878 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:50.077474 17793 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.081259 20237 tablet_copy_client.cc:308] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:50.082415 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet 5682259bc54c469e9c6a08723c1bbee0 from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:48881)
I20240501 20:10:50.082563 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet 5682259bc54c469e9c6a08723c1bbee0 from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:48881: session id = 047275bda1074ac5a63325e7e1109c61-5682259bc54c469e9c6a08723c1bbee0
I20240501 20:10:50.083174 16136 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.085652 16156 tablet_copy_source_session.cc:205] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:50.087345 20237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5682259bc54c469e9c6a08723c1bbee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.089280 15728 catalog_manager.cc:5484] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.114113 16004 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.134007 20109 raft_consensus.cc:1050] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:50.134351 20237 tablet_copy_client.cc:783] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:50.134860 20237 tablet_copy_client.cc:647] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:50.134907 20109 consensus_queue.cc:237] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:50.144220 15873 raft_consensus.cc:1261] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:50.144874 19797 consensus_queue.cc:1035] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:50.147064 17792 raft_consensus.cc:1261] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:50.147665 19791 consensus_queue.cc:1035] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:50.149019 20237 tablet_copy_client.cc:523] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:50.151226 16005 raft_consensus.cc:1261] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:50.152177 19797 consensus_queue.cc:1035] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:50.160617 20237 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:50.295194 15875 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:50.299340 19797 raft_consensus.cc:2932] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.306767 17793 raft_consensus.cc:2932] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.312146 15728 catalog_manager.cc:5484] T ff11a8e00b544b4193a2824541668492 P 9a0f0037700246bdb65e05aef95fb2be reported cstate change: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.316082 16003 raft_consensus.cc:2932] T ff11a8e00b544b4193a2824541668492 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.324412 15874 raft_consensus.cc:2932] T ff11a8e00b544b4193a2824541668492 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.333423 20236 tablet_replica.cc:330] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:50.333647 20236 raft_consensus.cc:2229] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:50.333797 20236 pending_rounds.cc:62] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: Trying to abort 1 pending ops.
I20240501 20:10:50.333887 20236 pending_rounds.cc:68] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022978046874869760 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c538581c71f646b5bc8d71cb6981f201" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } } }
I20240501 20:10:50.334306 20236 raft_consensus.cc:2894] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 20:10:50.334419 20236 raft_consensus.cc:2866] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:50.334523 20236 raft_consensus.cc:2258] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:50.337657 20236 ts_tablet_manager.cc:1875] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:50.348564 17927 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:50.349519 19031 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:50.351935 17927 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:50.352308 20236 ts_tablet_manager.cc:1888] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:50.352419 20236 log.cc:1196] T c538581c71f646b5bc8d71cb6981f201 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/c538581c71f646b5bc8d71cb6981f201
I20240501 20:10:50.368520 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet c538581c71f646b5bc8d71cb6981f201 (table rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]) successfully deleted
I20240501 20:10:50.399502 18040 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3eb13093b546d38fc71368f2083878 with delete_type TABLET_DATA_TOMBSTONED (TS cad00548f33849e7a19cecd760e1e429 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35338
W20240501 20:10:50.408720 20028 log.cc:924] Time spent T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20240501 20:10:50.451922 15873 consensus_queue.cc:237] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:50.454845 16134 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:50.455412 19522 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:50.457444 20073 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.458081 16134 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.471993 15728 catalog_manager.cc:5484] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 7 to 8, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:50.478780 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3eb13093b546d38fc71368f2083878 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:50.479367 20068 tablet_bootstrap.cc:492] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:50.479822 20068 tablet_bootstrap.cc:492] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:50.480212 20068 ts_tablet_manager.cc:1367] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.835s	user 0.048s	sys 0.001s
I20240501 20:10:50.481086 20068 raft_consensus.cc:348] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:50.481438 20068 raft_consensus.cc:729] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:50.481695 20068 consensus_queue.cc:260] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:50.489131 20068 ts_tablet_manager.cc:1398] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.006s
I20240501 20:10:50.489846 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-84d7bbca7af34a2db637b55635fa2f7a received from {username='slave'} at 127.8.68.200:48881
I20240501 20:10:50.489974 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-84d7bbca7af34a2db637b55635fa2f7a on tablet 84d7bbca7af34a2db637b55635fa2f7a with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:50.489961 17793 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:50.490738 19522 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:50.511584 20237 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:50.512022 20237 tablet_bootstrap.cc:492] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:50.512310 20237 ts_tablet_manager.cc:1367] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.352s	user 0.055s	sys 0.023s
I20240501 20:10:50.513267 20237 raft_consensus.cc:348] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:50.513579 20237 raft_consensus.cc:729] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:50.513864 20237 consensus_queue.cc:260] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:50.514535 20237 ts_tablet_manager.cc:1398] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:10:50.515164 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-5682259bc54c469e9c6a08723c1bbee0 received from {username='slave'} at 127.8.68.200:48881
I20240501 20:10:50.515297 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-5682259bc54c469e9c6a08723c1bbee0 on tablet 5682259bc54c469e9c6a08723c1bbee0 with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:50.520418 18190 raft_consensus.cc:1203] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240501 20:10:50.522356 15721 catalog_manager.cc:4797] TS cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): delete failed for tablet 8f3eb13093b546d38fc71368f2083878 because the tablet was not found. No further retry: Not found: Tablet not found: 8f3eb13093b546d38fc71368f2083878
I20240501 20:10:50.524490 20105 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:50.524924 20105 tablet_bootstrap.cc:492] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:50.525213 20105 ts_tablet_manager.cc:1367] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 1.390s	user 0.046s	sys 0.007s
I20240501 20:10:50.528455 18190 raft_consensus.cc:1203] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:50.525900 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f3eb13093b546d38fc71368f2083878 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:50.529132 19857 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:50.529505 19857 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:50.529771 19857 ts_tablet_manager.cc:1367] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 1.445s	user 0.054s	sys 0.003s
I20240501 20:10:50.538074 20105 raft_consensus.cc:348] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:50.538375 20105 raft_consensus.cc:729] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:50.538576 20105 consensus_queue.cc:260] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:50.540005 20105 ts_tablet_manager.cc:1398] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.000s
I20240501 20:10:50.547453 19857 raft_consensus.cc:348] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:50.547708 19857 raft_consensus.cc:729] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:50.546983 17793 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:50.547930 19857 consensus_queue.cc:260] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:50.548547 19857 ts_tablet_manager.cc:1398] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.019s	user 0.003s	sys 0.000s
I20240501 20:10:50.550143 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-3643d6995910404fb8a7e6ef9c635f42 received from {username='slave'} at 127.8.68.197:55615
I20240501 20:10:50.550261 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-3643d6995910404fb8a7e6ef9c635f42 on tablet 3643d6995910404fb8a7e6ef9c635f42 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:50.557335 20271 tablet_replica.cc:330] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:50.557545 20271 raft_consensus.cc:2229] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:50.557958 20271 raft_consensus.cc:2258] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:50.571216 16026 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-f638f341ff2341dc82889334eb87e3d0 received from {username='slave'} at 127.8.68.197:39757
I20240501 20:10:50.571341 16026 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-f638f341ff2341dc82889334eb87e3d0 on tablet f638f341ff2341dc82889334eb87e3d0 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:50.585537 20271 ts_tablet_manager.cc:1875] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:10:50.589347 20208 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.041s	sys 0.149s
W20240501 20:10:50.589484 20208 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.041s	sys 0.149s
I20240501 20:10:50.592134 20028 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:50.592670 20028 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:50.592941 20028 ts_tablet_manager.cc:1367] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 1.560s	user 0.039s	sys 0.006s
I20240501 20:10:50.598737 20028 raft_consensus.cc:348] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:50.599360 20028 raft_consensus.cc:729] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:50.599717 20028 consensus_queue.cc:260] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:50.600735 20028 ts_tablet_manager.cc:1398] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20240501 20:10:50.601869 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session d3917663631a4479b15355381486591c-f345303c72774848bb60e806cff22d12 received from {username='slave'} at 127.8.68.198:48071
I20240501 20:10:50.601985 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session d3917663631a4479b15355381486591c-f345303c72774848bb60e806cff22d12 on tablet f345303c72774848bb60e806cff22d12 with peer d3917663631a4479b15355381486591c
I20240501 20:10:50.603919 17929 raft_consensus.cc:1203] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:10:50.643656 17793 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:50.688405 20271 ts_tablet_manager.cc:1888] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 20:10:50.688637 20271 log.cc:1196] T 8f3eb13093b546d38fc71368f2083878 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/8f3eb13093b546d38fc71368f2083878
I20240501 20:10:50.689426 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet 8f3eb13093b546d38fc71368f2083878 (table rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]) successfully deleted
I20240501 20:10:50.691141 20208 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:50.691509 20208 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:50.691780 20208 ts_tablet_manager.cc:1367] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.856s	user 0.091s	sys 0.151s
I20240501 20:10:50.692541 20208 raft_consensus.cc:348] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:50.692816 20208 raft_consensus.cc:729] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:50.693002 20208 consensus_queue.cc:260] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:50.694329 20208 ts_tablet_manager.cc:1398] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:50.695072 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session d3917663631a4479b15355381486591c-3643d6995910404fb8a7e6ef9c635f42 received from {username='slave'} at 127.8.68.198:46329
I20240501 20:10:50.695202 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session d3917663631a4479b15355381486591c-3643d6995910404fb8a7e6ef9c635f42 on tablet 3643d6995910404fb8a7e6ef9c635f42 with peer d3917663631a4479b15355381486591c
I20240501 20:10:50.779788 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:50.802316 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:50.811062 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:50.826543 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:50.839190 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:50.846064 17793 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:50.872793 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:50.892374 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:50.909837 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:50.922981 19797 raft_consensus.cc:1050] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
I20240501 20:10:50.923919 19797 consensus_queue.cc:237] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:50.936048 18190 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:50.936473 17793 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:50.937543 19791 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:50.951318 16003 consensus_queue.cc:237] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:50.962132 19797 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:50.981034 17792 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:50.981707 19738 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:50.984196 19738 raft_consensus.cc:1013] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: attempt to promote peer 9a0f0037700246bdb65e05aef95fb2be: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:10:50.985442 18059 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:50.986380 20234 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:50.989742 16265 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:51.008596 19797 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:10:51.009864 16070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:38791, user_credentials={real_user=slave}} blocked reactor thread for 72480.5us
I20240501 20:10:51.013650 16134 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:51.019057 20234 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:51.023371 20234 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:51.027518 16134 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:51.028779 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f638f341ff2341dc82889334eb87e3d0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:51.030478 17793 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:51.032549 15728 catalog_manager.cc:5484] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 6 to 7, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
W20240501 20:10:51.034482 16069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:36919, user_credentials={real_user=slave}} blocked reactor thread for 44243.9us
I20240501 20:10:51.035168 17928 raft_consensus.cc:1203] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:10:51.051059 19791 raft_consensus.cc:1050] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
W20240501 20:10:51.051293 16071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:46677, user_credentials={real_user=slave}} blocked reactor thread for 53022.3us
I20240501 20:10:51.051921 19791 consensus_queue.cc:237] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:51.065301 20251 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.066828 18190 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.094427 16263 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.098491 17792 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.101930 15854 tablet_service.cc:1514] Processing DeleteTablet for tablet f638f341ff2341dc82889334eb87e3d0 with delete_type TABLET_DATA_TOMBSTONED (TS 51b81a5ae56b4dd99af1b0167c1da1e2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35708
I20240501 20:10:51.104077 15734 catalog_manager.cc:5484] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61 reported cstate change: config changed from index 7 to 8, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.109220 20356 tablet_replica.cc:330] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2: stopping tablet replica
I20240501 20:10:51.109959 20356 raft_consensus.cc:2229] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:51.110700 20356 raft_consensus.cc:2258] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:51.112355 20356 ts_tablet_manager.cc:1875] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:51.113756 16134 consensus_queue.cc:237] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:51.118712 17792 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.119465 20336 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.121012 20234 raft_consensus.cc:1050] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:51.121814 20234 consensus_queue.cc:237] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:51.126878 18190 raft_consensus.cc:1261] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.127635 16134 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:51.128118 20337 consensus_queue.cc:1035] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.128436 20234 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:51.129426 20336 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.129982 20356 ts_tablet_manager.cc:1888] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
W20240501 20:10:51.130204 19275 log.cc:924] Time spent T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20240501 20:10:51.130903 20356 log.cc:1196] T f638f341ff2341dc82889334eb87e3d0 P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/f638f341ff2341dc82889334eb87e3d0
I20240501 20:10:51.131484 19275 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:51.131925 19275 tablet_bootstrap.cc:492] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:51.131664 17795 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.132201 19275 ts_tablet_manager.cc:1367] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 1.732s	user 0.044s	sys 0.007s
I20240501 20:10:51.132925 19275 raft_consensus.cc:348] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:51.133153 19275 raft_consensus.cc:729] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:51.133350 19275 consensus_queue.cc:260] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:51.135650 15719 catalog_manager.cc:4830] TS 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): tablet f638f341ff2341dc82889334eb87e3d0 (table rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]) successfully deleted
I20240501 20:10:51.139735 17793 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:51.141908 20361 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:51.145586 19738 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:51.149854 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5682259bc54c469e9c6a08723c1bbee0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:51.151508 16134 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:51.153772 15727 catalog_manager.cc:5484] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:51.156200 17792 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:51.170038 15727 catalog_manager.cc:5484] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:51.172176 18189 raft_consensus.cc:1261] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:51.172950 19797 consensus_queue.cc:1035] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:51.175105 16003 raft_consensus.cc:1261] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:51.175751 20336 consensus_queue.cc:1035] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:51.176853 18188 raft_consensus.cc:2932] T 5682259bc54c469e9c6a08723c1bbee0 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.178874 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet 5682259bc54c469e9c6a08723c1bbee0 with delete_type TABLET_DATA_TOMBSTONED (TS f856e0d1074b431eb19763847958b89e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44866
I20240501 20:10:51.191346 15875 raft_consensus.cc:1261] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:51.191897 20336 consensus_queue.cc:1035] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:51.193601 18058 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:51.194525 19738 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:51.196019 18058 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:51.220150 20251 raft_consensus.cc:1050] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:51.220938 20251 consensus_queue.cc:237] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:51.235021 19275 ts_tablet_manager.cc:1398] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.103s	user 0.000s	sys 0.003s
I20240501 20:10:51.235850 16026 tablet_copy_service.cc:342] P 754fcd3b82f041568f546c6205f44272: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-f904b62401ff4528b3fa65dd4e11512a received from {username='slave'} at 127.8.68.199:37351
I20240501 20:10:51.235999 16026 tablet_copy_service.cc:433] P 754fcd3b82f041568f546c6205f44272: ending tablet copy session cad00548f33849e7a19cecd760e1e429-f904b62401ff4528b3fa65dd4e11512a on tablet f904b62401ff4528b3fa65dd4e11512a with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:51.243132 20337 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.245618 18190 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.247010 16004 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.253646 15875 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.253525 20366 tablet_replica.cc:330] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e: stopping tablet replica
I20240501 20:10:51.258786 20366 raft_consensus.cc:2229] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:51.258520 15731 catalog_manager.cc:5484] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61 reported cstate change: config changed from index 4 to 5, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.260223 20366 raft_consensus.cc:2258] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:51.263034 16004 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:51.264019 20336 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:51.265213 20366 ts_tablet_manager.cc:1875] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:51.265986 16134 consensus_queue.cc:237] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:51.268419 15873 raft_consensus.cc:1261] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:51.269177 17929 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:51.269228 16265 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:51.269937 19797 consensus_queue.cc:1035] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:51.272017 20336 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:51.272305 20327 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:51.273063 20328 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.276927 18190 raft_consensus.cc:1261] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:51.278198 19797 consensus_queue.cc:1035] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:51.278311 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84d7bbca7af34a2db637b55635fa2f7a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:51.284255 15729 catalog_manager.cc:5484] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:51.286993 18190 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.290478 15875 raft_consensus.cc:2932] T 84d7bbca7af34a2db637b55635fa2f7a P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:51.294032 19031 raft_consensus.cc:1050] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:51.294569 15986 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d7bbca7af34a2db637b55635fa2f7a with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:51.295100 19031 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:51.307675 20251 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.312335 17929 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.312460 16004 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.320480 20388 tablet_replica.cc:330] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:51.320886 20388 raft_consensus.cc:2229] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:51.324165 20388 raft_consensus.cc:2258] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:51.326730 20388 ts_tablet_manager.cc:1875] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:51.328548 20366 ts_tablet_manager.cc:1888] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:51.329988 17929 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.330230 16134 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.330835 18059 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.331722 17792 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.330899 20159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.333781 20159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.334199 20159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.334497 19670 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.341014 20388 ts_tablet_manager.cc:1888] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:51.344744 19670 raft_consensus.cc:1013] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: attempt to promote peer d3917663631a4479b15355381486591c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:10:51.348538 20366 log.cc:1196] T 5682259bc54c469e9c6a08723c1bbee0 P f856e0d1074b431eb19763847958b89e: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/5682259bc54c469e9c6a08723c1bbee0
I20240501 20:10:51.356142 16265 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.360291 15718 catalog_manager.cc:4830] TS f856e0d1074b431eb19763847958b89e (127.8.68.196:36919): tablet 5682259bc54c469e9c6a08723c1bbee0 (table rebalance_test_table_1 [id=9397d415e86d427d8fe2f9e906b59e2c]) successfully deleted
I20240501 20:10:51.363958 20388 log.cc:1196] T 84d7bbca7af34a2db637b55635fa2f7a P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/84d7bbca7af34a2db637b55635fa2f7a
I20240501 20:10:51.374374 18058 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:51.376216 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet 84d7bbca7af34a2db637b55635fa2f7a (table rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]) successfully deleted
I20240501 20:10:51.376188 15734 catalog_manager.cc:5484] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c reported cstate change: config changed from index 4 to 5, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.379370 20159 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:51.381398 16134 consensus_queue.cc:237] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:51.383494 16004 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:51.383183 17795 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:51.384056 20336 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:51.387740 20251 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.388370 16004 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.391220 17929 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:51.398190 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f345303c72774848bb60e806cff22d12 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:51.399291 20159 raft_consensus.cc:1050] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:51.400290 20159 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:51.400373 16134 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:51.402968 17793 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:51.403642 19670 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:51.405618 16136 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:51.398851 15734 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c reported cstate change: config changed from index 8 to 9, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:51.407119 17929 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:51.407644 19670 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:51.408056 19670 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:51.412233 15731 catalog_manager.cc:5484] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:51.413074 16134 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.416266 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet f345303c72774848bb60e806cff22d12 with delete_type TABLET_DATA_TOMBSTONED (TS f856e0d1074b431eb19763847958b89e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44866
I20240501 20:10:51.416586 20366 tablet_replica.cc:330] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e: stopping tablet replica
I20240501 20:10:51.416777 20366 raft_consensus.cc:2229] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:51.416983 20366 raft_consensus.cc:2258] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:51.417104 19670 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.418040 20366 ts_tablet_manager.cc:1875] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:51.422050 17795 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.425216 20366 ts_tablet_manager.cc:1888] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:51.425756 20366 log.cc:1196] T f345303c72774848bb60e806cff22d12 P f856e0d1074b431eb19763847958b89e: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/f345303c72774848bb60e806cff22d12
I20240501 20:10:51.426308 17929 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.426795 15718 catalog_manager.cc:4830] TS f856e0d1074b431eb19763847958b89e (127.8.68.196:36919): tablet f345303c72774848bb60e806cff22d12 (table rebalance_test_table_5 [id=623ab7f13c234a3ab8bf37522d87130c]) successfully deleted
I20240501 20:10:51.431157 15727 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c reported cstate change: config changed from index 9 to 10, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.437558 16265 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:51.440315 18060 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:51.441756 18060 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:51.442787 17929 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:51.443094 19031 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.443691 17792 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:51.444041 19031 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:51.444350 19670 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:51.444613 18060 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:51.449398 18060 raft_consensus.cc:858] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }: 
  New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
W20240501 20:10:51.449671 18060 raft_consensus.cc:1565] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 11' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }: 
  New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:51.449800 17928 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:51.450400 20251 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:51.451856 17928 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.450347 18060 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.463594 15875 consensus_queue.cc:237] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:51.472709 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:51.479280 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:51.480669 15811 consensus_peers.cc:485] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3eb13093b546d38fc71368f2083878. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:51.481555 16134 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.482488 19522 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.499275 19522 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:51.502990 19670 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.503572 16134 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:51.507699 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3643d6995910404fb8a7e6ef9c635f42 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:51.511775 17792 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.522010 15727 catalog_manager.cc:5484] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 8 to 9, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:51.517872 15731 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 10 to 11, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:51.524298 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:51.526500 17795 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.527457 19522 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.533944 17795 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:51.533987 17929 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.540511 18060 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.553160 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 3643d6995910404fb8a7e6ef9c635f42 with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:51.554913 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:51.561674 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:51.580005 18060 raft_consensus.cc:1203] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 20:10:51.581436 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:51.584432 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:51.585075 20436 tablet_replica.cc:330] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:51.585379 20436 raft_consensus.cc:2229] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:51.585701 20436 raft_consensus.cc:2258] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:51.587437 20436 ts_tablet_manager.cc:1875] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:51.601835 20436 ts_tablet_manager.cc:1888] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 20:10:51.602025 20436 log.cc:1196] T 3643d6995910404fb8a7e6ef9c635f42 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/3643d6995910404fb8a7e6ef9c635f42
I20240501 20:10:51.603046 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet 3643d6995910404fb8a7e6ef9c635f42 (table rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]) successfully deleted
I20240501 20:10:51.620599 16265 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:51.623636 17795 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:10:51.625581 20400 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:10:51.629751 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:51.642617 20400 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.647143 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3643d6995910404fb8a7e6ef9c635f42 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:51.649006 17795 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.649185 15728 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 11 to 12, VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:51.653383 18040 tablet_service.cc:1514] Processing DeleteTablet for tablet 3643d6995910404fb8a7e6ef9c635f42 with delete_type TABLET_DATA_TOMBSTONED (TS cad00548f33849e7a19cecd760e1e429 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:35338
I20240501 20:10:51.665474 16265 consensus_queue.cc:237] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:51.674098 17928 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:10:51.675192 20400 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:10:51.676579 17928 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:51.682193 18190 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:51.682998 15875 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:10:51.683629 16199 consensus_peers.cc:485] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98eb95ded28d433aad9523586e29947f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:51.683883 20417 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:51.684454 20417 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:51.698261 20400 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:51.702909 18190 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:51.704911 15727 catalog_manager.cc:5484] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:51.719019 20444 tablet_replica.cc:330] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: stopping tablet replica
I20240501 20:10:51.719720 20444 raft_consensus.cc:2229] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:51.720396 20444 raft_consensus.cc:2258] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:51.721655 15875 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:51.742671 20444 ts_tablet_manager.cc:1875] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:51.808305 16004 consensus_queue.cc:237] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:51.810870 17795 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.811132 18060 raft_consensus.cc:1261] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.811458 19738 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.815151 20361 consensus_queue.cc:1035] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.818845 19738 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:51.823084 18060 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:51.824569 20444 ts_tablet_manager.cc:1888] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 20:10:51.824724 20444 log.cc:1196] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-6/wal/wals/3643d6995910404fb8a7e6ef9c635f42
I20240501 20:10:51.825506 15721 catalog_manager.cc:4830] TS cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): tablet 3643d6995910404fb8a7e6ef9c635f42 (table rebalance_test_table_9 [id=fa7a41060cc1492b8f99aac4eedf2e0f]) successfully deleted
I20240501 20:10:51.831631 15729 catalog_manager.cc:5484] T f638f341ff2341dc82889334eb87e3d0 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:51.835419 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet f638f341ff2341dc82889334eb87e3d0 with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:51.835742 20436 tablet_replica.cc:330] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:51.835930 20436 raft_consensus.cc:2229] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:51.836295 20436 raft_consensus.cc:2258] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:51.836766 17795 raft_consensus.cc:2932] T f638f341ff2341dc82889334eb87e3d0 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:51.837350 20436 ts_tablet_manager.cc:1875] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:51.838564 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f638f341ff2341dc82889334eb87e3d0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:51.843466 20436 ts_tablet_manager.cc:1888] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:51.843637 20436 log.cc:1196] T f638f341ff2341dc82889334eb87e3d0 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/f638f341ff2341dc82889334eb87e3d0
I20240501 20:10:51.845132 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet f638f341ff2341dc82889334eb87e3d0 (table rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]) successfully deleted
I20240501 20:10:51.859325 15875 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } }
I20240501 20:10:51.871994 17928 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.872648 19522 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.882016 18060 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.882356 16134 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:51.883380 19522 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.883704 20073 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:51.901327 19522 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } }
I20240501 20:10:51.907402 18058 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } }
I20240501 20:10:51.912734 17928 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } }
I20240501 20:10:51.917428 15727 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } }
I20240501 20:10:51.928791 15875 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:51.938338 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:51.938458 18060 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:51.938907 20073 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:51.941329 17928 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:51.941869 20073 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:51.955727 20491 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:51.955845 17929 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
W20240501 20:10:51.963471 15809 consensus_peers.cc:485] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e6b15e4aebf4b1f9a3176f87083703e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:51.962116 15727 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c reported cstate change: config changed from index 9 to 10, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:51.969902 15719 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2e6b15e4aebf4b1f9a3176f87083703e with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:10:51.996158 18060 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.001498 16136 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:52.002844 20491 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:52.005196 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:52.005759 16136 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } }
I20240501 20:10:52.011399 16136 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.015616 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:52.040285 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:52.095896 20500 ts_tablet_manager.cc:902] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:52.107671 20499 ts_tablet_manager.cc:902] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:52.110234 20500 tablet_copy_client.cc:308] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:52.111229 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet 8f3eb13093b546d38fc71368f2083878 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:37603)
I20240501 20:10:52.111384 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet 8f3eb13093b546d38fc71368f2083878 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:37603: session id = cad00548f33849e7a19cecd760e1e429-8f3eb13093b546d38fc71368f2083878
I20240501 20:10:52.114223 15895 tablet_copy_source_session.cc:205] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:52.114573 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:52.118412 20500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3eb13093b546d38fc71368f2083878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.147267 20500 tablet_copy_client.cc:783] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:52.147720 20500 tablet_copy_client.cc:647] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:52.147987 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:52.150789 20500 tablet_copy_client.cc:523] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:52.158622 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:52.182670 19738 raft_consensus.cc:1050] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:52.183477 19738 consensus_queue.cc:237] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:52.193432 18060 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:52.193989 20361 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:52.209503 18189 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:52.210168 20361 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:52.219355 16265 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:52.219905 20361 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:52.222848 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:52.242424 15875 consensus_queue.cc:237] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:52.256912 20361 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:52.266633 15729 catalog_manager.cc:5484] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.266985 17928 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:10:52.267369 15811 consensus_peers.cc:485] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c74a6da428b04df58b7edcdf88d81cca. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:52.267601 20490 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:52.270475 18060 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:52.274464 20500 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:52.276063 16004 consensus_queue.cc:237] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:52.279366 18190 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:52.280016 20466 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:52.280694 20493 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:52.281266 18190 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:52.284121 17928 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:52.288803 16134 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:52.291923 15729 catalog_manager.cc:5484] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.293274 19522 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:52.310557 16134 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:52.317183 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:52.328058 19738 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:52.331233 18189 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:52.334008 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f904b62401ff4528b3fa65dd4e11512a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:52.335316 15729 catalog_manager.cc:5484] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.340649 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet f904b62401ff4528b3fa65dd4e11512a with delete_type TABLET_DATA_TOMBSTONED (TS f856e0d1074b431eb19763847958b89e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44866
I20240501 20:10:52.353751 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:52.359954 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:52.396723 20549 ts_tablet_manager.cc:902] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:52.397174 18059 raft_consensus.cc:1261] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:52.399519 20531 consensus_queue.cc:1035] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:52.401249 18060 raft_consensus.cc:2932] T f904b62401ff4528b3fa65dd4e11512a P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f856e0d1074b431eb19763847958b89e (127.8.68.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:52.403159 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:52.415158 20542 tablet_replica.cc:330] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: stopping tablet replica
I20240501 20:10:52.415498 20542 raft_consensus.cc:2229] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:52.415742 20542 pending_rounds.cc:62] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: Trying to abort 1 pending ops.
I20240501 20:10:52.418751 20542 pending_rounds.cc:68] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022978056941527040 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f904b62401ff4528b3fa65dd4e11512a" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } } }
I20240501 20:10:52.419330 20542 raft_consensus.cc:2894] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 20:10:52.419579 20542 raft_consensus.cc:2866] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:52.419728 20542 raft_consensus.cc:2258] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:52.420693 20542 ts_tablet_manager.cc:1875] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:52.424542 20549 tablet_copy_client.cc:308] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:52.425223 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet 2e6b15e4aebf4b1f9a3176f87083703e from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:35897)
I20240501 20:10:52.425421 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet 2e6b15e4aebf4b1f9a3176f87083703e from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:35897: session id = 047275bda1074ac5a63325e7e1109c61-2e6b15e4aebf4b1f9a3176f87083703e
I20240501 20:10:52.428404 15895 tablet_copy_source_session.cc:205] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:52.430083 20549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6b15e4aebf4b1f9a3176f87083703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.434813 16134 consensus_queue.cc:237] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:52.443850 20499 tablet_copy_client.cc:308] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:52.444545 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet 98eb95ded28d433aad9523586e29947f from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:46329)
I20240501 20:10:52.444703 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet 98eb95ded28d433aad9523586e29947f from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:46329: session id = d3917663631a4479b15355381486591c-98eb95ded28d433aad9523586e29947f
I20240501 20:10:52.446527 20549 tablet_copy_client.cc:783] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:52.446751 20549 tablet_copy_client.cc:647] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:52.447676 16285 tablet_copy_source_session.cc:205] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:52.449162 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98eb95ded28d433aad9523586e29947f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.449420 20549 tablet_copy_client.cc:523] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:52.453440 20549 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:52.476567 20499 tablet_copy_client.cc:783] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:52.476874 20499 tablet_copy_client.cc:647] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:52.478523 20499 tablet_copy_client.cc:523] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:52.489655 20499 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: Bootstrap starting.
I20240501 20:10:52.500041 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:52.502182 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:52.505787 16004 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:52.505859 16265 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:52.507056 16070 consensus_peers.cc:485] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eef58fcca15041a39e685a7cdcb04f2b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:52.507579 19797 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:52.520902 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:52.523046 20542 ts_tablet_manager.cc:1888] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:52.523178 20542 log.cc:1196] T f904b62401ff4528b3fa65dd4e11512a P f856e0d1074b431eb19763847958b89e: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/f904b62401ff4528b3fa65dd4e11512a
I20240501 20:10:52.538141 15718 catalog_manager.cc:4830] TS f856e0d1074b431eb19763847958b89e (127.8.68.196:36919): tablet f904b62401ff4528b3fa65dd4e11512a (table rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]) successfully deleted
I20240501 20:10:52.550714 15875 consensus_queue.cc:237] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:52.551739 20567 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:52.554845 16004 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:52.555770 20490 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:10:52.557998 15809 consensus_peers.cc:485] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 627e9e5c120747f1916855ea83008657. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:52.558472 17795 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:52.559039 20490 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:52.567036 15875 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:52.567350 19522 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.571255 18060 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:52.572480 20493 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:10:52.574154 15809 consensus_peers.cc:485] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:52.575552 17795 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.577404 20490 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.578241 18060 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.581763 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e6b15e4aebf4b1f9a3176f87083703e with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:52.581722 15728 catalog_manager.cc:5484] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.583856 15729 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 10 to 11, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.590236 16004 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.606711 17908 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6b15e4aebf4b1f9a3176f87083703e with delete_type TABLET_DATA_TOMBSTONED (TS d3917663631a4479b15355381486591c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:58082
I20240501 20:10:52.608597 20579 tablet_replica.cc:330] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: stopping tablet replica
I20240501 20:10:52.609148 20579 raft_consensus.cc:2229] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:52.614742 20579 raft_consensus.cc:2258] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:52.626972 20579 ts_tablet_manager.cc:1875] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:10:52.630447 16071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.194:46677, user_credentials={real_user=slave}} blocked reactor thread for 123983us
W20240501 20:10:52.630640 16069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.196:36919, user_credentials={real_user=slave}} blocked reactor thread for 124232us
I20240501 20:10:52.641815 16136 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:52.642876 20490 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:52.649318 19797 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.653295 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:52.662295 20500 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:52.662775 20500 tablet_bootstrap.cc:492] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:52.663058 20500 ts_tablet_manager.cc:1367] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.389s	user 0.038s	sys 0.018s
I20240501 20:10:52.663812 20500 raft_consensus.cc:348] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:52.664057 20500 raft_consensus.cc:729] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:52.663687 16263 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.664245 20500 consensus_queue.cc:260] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:52.664844 20500 ts_tablet_manager.cc:1398] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:10:52.666363 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-8f3eb13093b546d38fc71368f2083878 received from {username='slave'} at 127.8.68.199:37603
I20240501 20:10:52.666498 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session cad00548f33849e7a19cecd760e1e429-8f3eb13093b546d38fc71368f2083878 on tablet 8f3eb13093b546d38fc71368f2083878 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:52.671890 15728 catalog_manager.cc:5484] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.675853 20549 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:52.676396 20549 tablet_bootstrap.cc:492] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:52.676687 20549 ts_tablet_manager.cc:1367] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.223s	user 0.042s	sys 0.010s
I20240501 20:10:52.683434 20549 raft_consensus.cc:348] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:52.683724 20549 raft_consensus.cc:729] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:52.683926 20549 consensus_queue.cc:260] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:52.688329 16136 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.710777 20549 ts_tablet_manager.cc:1398] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.034s	user 0.001s	sys 0.001s
I20240501 20:10:52.711508 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-2e6b15e4aebf4b1f9a3176f87083703e received from {username='slave'} at 127.8.68.200:35897
I20240501 20:10:52.711648 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-2e6b15e4aebf4b1f9a3176f87083703e on tablet 2e6b15e4aebf4b1f9a3176f87083703e with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:52.729647 16004 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:52.741395 16263 consensus_queue.cc:237] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:52.751039 20579 ts_tablet_manager.cc:1888] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 20:10:52.751192 20579 log.cc:1196] T 2e6b15e4aebf4b1f9a3176f87083703e P d3917663631a4479b15355381486591c: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/wals/2e6b15e4aebf4b1f9a3176f87083703e
I20240501 20:10:52.752072 15718 catalog_manager.cc:4830] TS d3917663631a4479b15355381486591c (127.8.68.198:44869): tablet 2e6b15e4aebf4b1f9a3176f87083703e (table rebalance_test_table_6 [id=bdaa80eaeb6c4114930a1001be0cb61d]) successfully deleted
I20240501 20:10:52.756275 18190 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:52.757146 20400 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:10:52.764523 16199 consensus_peers.cc:485] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb90224d08f84a2faf7dcc063c6af56e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:52.766366 15875 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:52.766891 20400 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:52.772212 20500 ts_tablet_manager.cc:902] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:52.784199 20400 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:52.787937 15873 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:52.793109 15728 catalog_manager.cc:5484] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.813668 18189 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:52.818719 20500 tablet_copy_client.cc:308] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:52.819593 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet c74a6da428b04df58b7edcdf88d81cca from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:37603)
I20240501 20:10:52.819747 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet c74a6da428b04df58b7edcdf88d81cca from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:37603: session id = cad00548f33849e7a19cecd760e1e429-c74a6da428b04df58b7edcdf88d81cca
I20240501 20:10:52.823263 15895 tablet_copy_source_session.cc:205] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:52.826107 20500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74a6da428b04df58b7edcdf88d81cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.843482 20500 tablet_copy_client.cc:783] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:52.843708 20500 tablet_copy_client.cc:647] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:52.849018 20500 tablet_copy_client.cc:523] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:52.863397 20500 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:52.885617 15873 consensus_queue.cc:237] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: true } }
I20240501 20:10:52.887125 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:52.918900 17795 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 20:10:52.920053 15810 consensus_peers.cc:485] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Couldn't send request to peer 754fcd3b82f041568f546c6205f44272. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:52.920503 20493 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:52.928364 18190 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:52.928840 20493 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:52.938663 20073 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: true } } }
I20240501 20:10:52.939647 18190 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: true } } }
I20240501 20:10:52.945747 15727 catalog_manager.cc:5484] T ddf5ddac08364043bf2e27184d5d3213 P 047275bda1074ac5a63325e7e1109c61 reported cstate change: config changed from index 9 to 10, NON_VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: true } } }
I20240501 20:10:52.949424 20499 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:52.949944 20499 tablet_bootstrap.cc:492] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:52.950971 20499 ts_tablet_manager.cc:1367] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.461s	user 0.036s	sys 0.013s
I20240501 20:10:52.956182 18170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:52.956741 20499 raft_consensus.cc:348] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:52.962234 20499 raft_consensus.cc:729] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:52.963048 20499 consensus_queue.cc:260] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:52.971410 17795 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: true } } }
I20240501 20:10:52.999289 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:52.999792 20499 ts_tablet_manager.cc:1398] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.048s	user 0.002s	sys 0.000s
I20240501 20:10:53.000555 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session d3917663631a4479b15355381486591c-98eb95ded28d433aad9523586e29947f received from {username='slave'} at 127.8.68.198:46329
I20240501 20:10:53.000717 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session d3917663631a4479b15355381486591c-98eb95ded28d433aad9523586e29947f on tablet 98eb95ded28d433aad9523586e29947f with peer d3917663631a4479b15355381486591c
I20240501 20:10:53.067880 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:53.087240 16136 consensus_queue.cc:237] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:53.098724 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:53.103855 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:53.105031 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:53.107318 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:53.111050 18060 raft_consensus.cc:1203] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240501 20:10:53.114538 20549 ts_tablet_manager.cc:902] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
W20240501 20:10:53.143710 16072 consensus_peers.cc:485] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f345303c72774848bb60e806cff22d12. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:53.144716 17928 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:53.145330 16004 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:53.145318 20566 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:53.145936 19797 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:53.183528 20660 ts_tablet_manager.cc:902] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:53.183583 19797 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:53.186326 17929 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:53.190620 15728 catalog_manager.cc:5484] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:53.192225 18190 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:53.196442 16004 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:53.210513 20660 tablet_copy_client.cc:308] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:53.211417 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet 627e9e5c120747f1916855ea83008657 from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:35897)
I20240501 20:10:53.211618 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet 627e9e5c120747f1916855ea83008657 from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:35897: session id = 047275bda1074ac5a63325e7e1109c61-627e9e5c120747f1916855ea83008657
I20240501 20:10:53.218712 15895 tablet_copy_source_session.cc:205] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:53.220194 20660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627e9e5c120747f1916855ea83008657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.229738 20660 tablet_copy_client.cc:783] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:53.230124 20660 tablet_copy_client.cc:647] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:53.232300 20660 tablet_copy_client.cc:523] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:53.239693 20660 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:53.249714 20549 tablet_copy_client.cc:308] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:53.251106 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet eef58fcca15041a39e685a7cdcb04f2b from peer 047275bda1074ac5a63325e7e1109c61 ({username='slave'} at 127.8.68.200:48881)
I20240501 20:10:53.251257 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet eef58fcca15041a39e685a7cdcb04f2b from peer 047275bda1074ac5a63325e7e1109c61 at {username='slave'} at 127.8.68.200:48881: session id = 047275bda1074ac5a63325e7e1109c61-eef58fcca15041a39e685a7cdcb04f2b
I20240501 20:10:53.254117 16156 tablet_copy_source_session.cc:205] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:53.255396 20549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef58fcca15041a39e685a7cdcb04f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.268760 16263 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
W20240501 20:10:53.271579 16202 consensus_peers.cc:485] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:53.272815 17795 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:10:53.273428 20159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:10:53.275174 17928 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:10:53.275763 20159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:10:53.293330 20400 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:53.293917 20549 tablet_copy_client.cc:783] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:53.294135 20549 tablet_copy_client.cc:647] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:53.297446 16136 consensus_queue.cc:237] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:53.297801 20500 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:53.300529 15727 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 12 to 13, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:53.301434 16004 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:53.303256 20499 ts_tablet_manager.cc:902] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:53.304014 16263 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:53.305480 20566 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:53.307098 19797 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:53.307338 20500 tablet_bootstrap.cc:492] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:53.307612 20500 ts_tablet_manager.cc:1367] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.444s	user 0.044s	sys 0.010s
W20240501 20:10:53.307969 16070 consensus_peers.cc:485] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:53.308406 16072 consensus_peers.cc:485] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eef58fcca15041a39e685a7cdcb04f2b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:53.313349 17928 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:53.314615 19797 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:53.315814 20500 raft_consensus.cc:348] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:53.316067 20500 raft_consensus.cc:729] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:53.316291 20500 consensus_queue.cc:260] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:53.316849 20500 ts_tablet_manager.cc:1398] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.000s
I20240501 20:10:53.319890 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-c74a6da428b04df58b7edcdf88d81cca received from {username='slave'} at 127.8.68.199:37603
I20240501 20:10:53.320068 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session cad00548f33849e7a19cecd760e1e429-c74a6da428b04df58b7edcdf88d81cca on tablet c74a6da428b04df58b7edcdf88d81cca with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:53.321285 16263 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:53.326644 16004 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:53.331077 17792 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:53.341392 20549 tablet_copy_client.cc:523] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:53.345943 15731 catalog_manager.cc:5484] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:53.348829 20549 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: Bootstrap starting.
I20240501 20:10:53.352550 20499 tablet_copy_client.cc:308] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:53.354032 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet eb90224d08f84a2faf7dcc063c6af56e from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:46329)
I20240501 20:10:53.354211 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet eb90224d08f84a2faf7dcc063c6af56e from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:46329: session id = d3917663631a4479b15355381486591c-eb90224d08f84a2faf7dcc063c6af56e
I20240501 20:10:53.357074 16285 tablet_copy_source_session.cc:205] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:53.360268 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb90224d08f84a2faf7dcc063c6af56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.370637 20710 ts_tablet_manager.cc:902] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:53.404863 20710 tablet_copy_client.cc:308] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:53.406196 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet ddf5ddac08364043bf2e27184d5d3213 from peer 754fcd3b82f041568f546c6205f44272 ({username='slave'} at 127.8.68.194:49815)
I20240501 20:10:53.406379 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet ddf5ddac08364043bf2e27184d5d3213 from peer 754fcd3b82f041568f546c6205f44272 at {username='slave'} at 127.8.68.194:49815: session id = 754fcd3b82f041568f546c6205f44272-ddf5ddac08364043bf2e27184d5d3213
I20240501 20:10:53.409790 15895 tablet_copy_source_session.cc:205] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:53.411579 20710 ts_tablet_manager.cc:1875] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:10:53.413591 15873 consensus_queue.cc:237] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:53.419811 20499 tablet_copy_client.cc:783] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:53.420060 20499 tablet_copy_client.cc:647] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:53.423691 20499 tablet_copy_client.cc:523] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 20:10:53.424912 15809 consensus_peers.cc:485] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e483217b8761457d84edc49253877719. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:53.425186 20710 ts_tablet_manager.cc:1888] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 20:10:53.425477 20710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf5ddac08364043bf2e27184d5d3213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.426075 17928 raft_consensus.cc:1261] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:53.426817 19522 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:53.427115 17928 raft_consensus.cc:1203] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 20:10:53.433744 16136 raft_consensus.cc:1261] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:53.434954 19522 consensus_queue.cc:1035] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:53.440074 20073 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:53.441296 20710 tablet_copy_client.cc:783] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:53.441619 20710 tablet_copy_client.cc:647] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:53.443473 16136 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:53.444862 20710 tablet_copy_client.cc:523] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:53.453342 15731 catalog_manager.cc:5484] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:53.456691 20710 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Bootstrap starting.
I20240501 20:10:53.472023 15873 consensus_queue.cc:237] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:53.479306 20499 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: Bootstrap starting.
W20240501 20:10:53.481997 15808 consensus_peers.cc:485] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d48de8e0c12b432ea5079390400302c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:53.482041 18190 raft_consensus.cc:1261] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:53.482651 20073 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:53.484702 16263 raft_consensus.cc:1261] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:53.485978 20073 consensus_queue.cc:1035] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:53.512554 20549 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:53.513039 20549 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:53.513299 20549 ts_tablet_manager.cc:1367] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.165s	user 0.051s	sys 0.006s
I20240501 20:10:53.513861 19522 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:53.514195 20549 raft_consensus.cc:348] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:53.514428 20549 raft_consensus.cc:729] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:53.514658 20549 consensus_queue.cc:260] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:53.516104 20549 ts_tablet_manager.cc:1398] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:10:53.516718 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-eef58fcca15041a39e685a7cdcb04f2b received from {username='slave'} at 127.8.68.200:48881
I20240501 20:10:53.516865 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-eef58fcca15041a39e685a7cdcb04f2b on tablet eef58fcca15041a39e685a7cdcb04f2b with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:53.518182 16265 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:53.523813 15733 catalog_manager.cc:5484] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:53.527299 20073 raft_consensus.cc:1050] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER 047275bda1074ac5a63325e7e1109c61 to VOTER
I20240501 20:10:53.528024 20073 consensus_queue.cc:237] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:53.530938 16136 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:10:53.531894 18058 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:10:53.532572 20493 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:10:53.535144 18189 raft_consensus.cc:1261] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:10:53.535193 20749 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:10:53.535768 20493 consensus_queue.cc:1035] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:10:53.537551 20490 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:53.540226 18189 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:53.542014 18058 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:53.547492 15727 catalog_manager.cc:5484] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 11 to 12, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:53.547649 17928 raft_consensus.cc:2932] T e483217b8761457d84edc49253877719 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:53.548022 16136 raft_consensus.cc:2932] T 2e6b15e4aebf4b1f9a3176f87083703e P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } }
I20240501 20:10:53.576860 20660 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:53.583019 20660 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: Bootstrap complete.
I20240501 20:10:53.583292 20660 ts_tablet_manager.cc:1367] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: Time spent bootstrapping tablet: real 0.344s	user 0.047s	sys 0.004s
I20240501 20:10:53.589371 20660 raft_consensus.cc:348] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:53.589641 20660 raft_consensus.cc:729] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 047275bda1074ac5a63325e7e1109c61, State: Initialized, Role: LEARNER
I20240501 20:10:53.589826 20660 consensus_queue.cc:260] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
I20240501 20:10:53.601295 18189 raft_consensus.cc:2932] T d48de8e0c12b432ea5079390400302c1 P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:53.603073 20660 ts_tablet_manager.cc:1398] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.000s
I20240501 20:10:53.605147 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 047275bda1074ac5a63325e7e1109c61-627e9e5c120747f1916855ea83008657 received from {username='slave'} at 127.8.68.200:35897
I20240501 20:10:53.605306 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 047275bda1074ac5a63325e7e1109c61-627e9e5c120747f1916855ea83008657 on tablet 627e9e5c120747f1916855ea83008657 with peer 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:53.636657 20765 ts_tablet_manager.cc:902] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:53.643100 18189 raft_consensus.cc:1203] T 627e9e5c120747f1916855ea83008657 P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 20:10:53.646628 20710 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:53.647120 20710 tablet_bootstrap.cc:492] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Bootstrap complete.
I20240501 20:10:53.648615 20710 ts_tablet_manager.cc:1367] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Time spent bootstrapping tablet: real 0.192s	user 0.051s	sys 0.006s
I20240501 20:10:53.656106 20710 raft_consensus.cc:348] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: true } }
I20240501 20:10:53.656420 20710 raft_consensus.cc:729] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 754fcd3b82f041568f546c6205f44272, State: Initialized, Role: LEARNER
I20240501 20:10:53.656669 20710 consensus_queue.cc:260] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: true } }
I20240501 20:10:53.657320 20710 ts_tablet_manager.cc:1398] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.000s
I20240501 20:10:53.659058 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 754fcd3b82f041568f546c6205f44272-ddf5ddac08364043bf2e27184d5d3213 received from {username='slave'} at 127.8.68.194:49815
I20240501 20:10:53.659194 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 754fcd3b82f041568f546c6205f44272-ddf5ddac08364043bf2e27184d5d3213 on tablet ddf5ddac08364043bf2e27184d5d3213 with peer 754fcd3b82f041568f546c6205f44272
I20240501 20:10:53.677397 16263 consensus_queue.cc:237] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:53.683118 16136 raft_consensus.cc:1261] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:10:53.683492 16199 consensus_peers.cc:485] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3ba52a871714255aa1bd34d9daca920. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:53.683887 20159 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:53.686755 18058 raft_consensus.cc:1261] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:53.688269 20702 consensus_queue.cc:1035] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:53.703408 15873 consensus_queue.cc:237] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:53.703707 20499 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:53.704128 20499 tablet_bootstrap.cc:492] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: Bootstrap complete.
I20240501 20:10:53.704396 20499 ts_tablet_manager.cc:1367] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: Time spent bootstrapping tablet: real 0.225s	user 0.036s	sys 0.012s
I20240501 20:10:53.705142 20499 raft_consensus.cc:348] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:53.705358 20499 raft_consensus.cc:729] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3917663631a4479b15355381486591c, State: Initialized, Role: LEARNER
I20240501 20:10:53.705761 20765 tablet_copy_client.cc:308] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:53.706022 20499 consensus_queue.cc:260] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:53.706915 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet f345303c72774848bb60e806cff22d12 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:53423)
I20240501 20:10:53.707083 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet f345303c72774848bb60e806cff22d12 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:53423: session id = 9a0f0037700246bdb65e05aef95fb2be-f345303c72774848bb60e806cff22d12
I20240501 20:10:53.709518 20499 ts_tablet_manager.cc:1398] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.004s
I20240501 20:10:53.709815 16156 tablet_copy_source_session.cc:205] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:53.711079 20765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f345303c72774848bb60e806cff22d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.712059 20073 raft_consensus.cc:1050] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:53.712881 20073 consensus_queue.cc:237] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:53.716749 16134 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:53.717667 17795 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:53.718606 20493 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:53.720324 20765 tablet_copy_client.cc:783] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:53.720580 20765 tablet_copy_client.cc:647] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:53.720580 20752 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:53.721989 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session d3917663631a4479b15355381486591c-eb90224d08f84a2faf7dcc063c6af56e received from {username='slave'} at 127.8.68.198:46329
I20240501 20:10:53.722119 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session d3917663631a4479b15355381486591c-eb90224d08f84a2faf7dcc063c6af56e on tablet eb90224d08f84a2faf7dcc063c6af56e with peer d3917663631a4479b15355381486591c
I20240501 20:10:53.723296 18058 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:53.723826 20493 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:53.725020 20765 tablet_copy_client.cc:523] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:53.725574 20500 ts_tablet_manager.cc:902] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:53.727252 20753 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:53.728704 18059 raft_consensus.cc:1203] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 20:10:53.731374 17795 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:53.732079 18189 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:53.732627 20749 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:53.735280 16134 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:53.736828 20765 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:53.734223 15729 catalog_manager.cc:5484] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 9 to 10, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:53.739061 16004 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 20:10:53.742568 15811 consensus_peers.cc:485] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:53.746204 20493 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:53.750062 18188 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:53.752357 15729 catalog_manager.cc:5484] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 8 to 9, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:53.776984 20493 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:53.781838 20702 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:53.791170 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:53.805168 18059 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:53.811091 15729 catalog_manager.cc:5484] T d3ba52a871714255aa1bd34d9daca920 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:53.822211 16134 raft_consensus.cc:2932] T d3ba52a871714255aa1bd34d9daca920 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d3917663631a4479b15355381486591c (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } }
I20240501 20:10:53.834021 20765 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:53.834450 20765 tablet_bootstrap.cc:492] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:53.834756 20765 ts_tablet_manager.cc:1367] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.098s	user 0.042s	sys 0.011s
I20240501 20:10:53.835700 20159 raft_consensus.cc:1050] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:53.835523 20765 raft_consensus.cc:348] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:53.835798 20765 raft_consensus.cc:729] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:53.835987 20765 consensus_queue.cc:260] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:53.836445 20159 consensus_queue.cc:237] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:53.839700 20808 ts_tablet_manager.cc:902] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:53.843358 20765 ts_tablet_manager.cc:1398] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240501 20:10:53.844314 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-f345303c72774848bb60e806cff22d12 received from {username='slave'} at 127.8.68.197:53423
I20240501 20:10:53.844436 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-f345303c72774848bb60e806cff22d12 on tablet f345303c72774848bb60e806cff22d12 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:53.847108 18188 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:53.847823 20400 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:53.850891 17928 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:53.851485 20400 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:53.858583 15873 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:53.859151 20400 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:53.861327 20808 tablet_copy_client.cc:308] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:53.861477 20400 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:53.861985 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet eef58fcca15041a39e685a7cdcb04f2b from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:53283)
I20240501 20:10:53.862144 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet eef58fcca15041a39e685a7cdcb04f2b from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:53283: session id = cad00548f33849e7a19cecd760e1e429-eef58fcca15041a39e685a7cdcb04f2b
I20240501 20:10:53.862886 17928 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:53.865078 16156 tablet_copy_source_session.cc:205] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:53.866330 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef58fcca15041a39e685a7cdcb04f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.873665 15728 catalog_manager.cc:5484] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:53.876343 18188 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:53.890606 20660 ts_tablet_manager.cc:902] T e483217b8761457d84edc49253877719 P 047275bda1074ac5a63325e7e1109c61: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:53.890647 17929 raft_consensus.cc:1203] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
F20240501 20:10:53.890877 20660 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0f25f3a86f0b494b89fe8f86aae4f4cd.queue_time_us, tablet-download-pool-0f25f3a86f0b494b89fe8f86aae4f4cd.queue_time_us) (tablet-download-pool-0f25f3a86f0b494b89fe8f86aae4f4cd.run_cpu_time_us, tablet-download-pool-0f25f3a86f0b494b89fe8f86aae4f4cd.run_cpu_time_us) (tablet-download-pool-0f25f3a86f0b494b89fe8f86aae4f4cd.run_wall_time_us, tablet-download-pool-0f25f3a86f0b494b89fe8f86aae4f4cd.run_wall_time_us) (tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.queue_time_us, tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.queue_time_us) (tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.run_cpu_time_us, tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.run_cpu_time_us) (tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.run_wall_time_us, tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.run_wall_time_us) (tablet-download-pool-5682259bc54c469e9c6a08723c1bbee0.queue_time_us, tablet-download-pool-5682259bc54c469e9c6a08723c1bbee0.queue_time_us) (tablet-download-pool-5682259bc54c469e9c6a08723c1bbee0.run_cpu_time_us, tablet-download-pool-5682259bc54c469e9c6a08723c1bbee0.run_cpu_time_us) (tablet-download-pool-5682259bc54c469e9c6a08723c1bbee0.run_wall_time_us, tablet-download-pool-5682259bc54c469e9c6a08723c1bbee0.run_wall_time_us) (tablet-download-pool-627e9e5c120747f1916855ea83008657.queue_time_us, tablet-download-pool-627e9e5c120747f1916855ea83008657.queue_time_us) (tablet-download-pool-627e9e5c120747f1916855ea83008657.run_cpu_time_us, tablet-download-pool-627e9e5c120747f1916855ea83008657.run_cpu_time_us) (tablet-download-pool-627e9e5c120747f1916855ea83008657.run_wall_time_us, tablet-download-pool-627e9e5c120747f1916855ea83008657.run_wall_time_us) (tablet-download-pool-84d7bbca7af34a2db637b55635fa2f7a.queue_time_us, tablet-download-pool-84d7bbca7af34a2db637b55635fa2f7a.queue_time_us) (tablet-download-pool-84d7bbca7af34a2db637b55635fa2f7a.run_cpu_time_us, tablet-download-pool-84d7bbca7af34a2db637b55635fa2f7a.run_cpu_time_us) (tablet-download-pool-84d7bbca7af34a2db637b55635fa2f7a.run_wall_time_us, tablet-download-pool-84d7bbca7af34a2db637b55635fa2f7a.run_wall_time_us) (tablet-download-pool-8e0746bed2cf43b9bc7765c270a8aa62.queue_time_us, tablet-download-pool-8e0746bed2cf43b9bc7765c270a8aa62.queue_time_us) (tablet-download-pool-8e0746bed2cf43b9bc7765c270a8aa62.run_cpu_time_us, tablet-download-pool-8e0746bed2cf43b9bc7765c270a8aa62.run_cpu_time_us) (tablet-download-pool-8e0746bed2cf43b9bc7765c270a8aa62.run_wall_time_us, tablet-download-pool-8e0746bed2cf43b9bc7765c270a8aa62.run_wall_time_us) (tablet-download-pool-98eb95ded28d433aad9523586e29947f.queue_time_us, tablet-download-pool-98eb95ded28d433aad9523586e29947f.queue_time_us) (tablet-download-pool-98eb95ded28d433aad9523586e29947f.run_cpu_time_us, tablet-download-pool-98eb95ded28d433aad9523586e29947f.run_cpu_time_us) (tablet-download-pool-98eb95ded28d433aad9523586e29947f.run_wall_time_us, tablet-download-pool-98eb95ded28d433aad9523586e29947f.run_wall_time_us) (tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.queue_time_us, tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.queue_time_us) (tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.run_cpu_time_us, tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.run_cpu_time_us) (tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.run_wall_time_us, tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.run_wall_time_us) (tablet-download-pool-d48de8e0c12b432ea5079390400302c1.queue_time_us, tablet-download-pool-d48de8e0c12b432ea5079390400302c1.queue_time_us) (tablet-download-pool-d48de8e0c12b432ea5079390400302c1.run_cpu_time_us, tablet-download-pool-d48de8e0c12b432ea5079390400302c1.run_cpu_time_us) (tablet-download-pool-d48de8e0c12b432ea5079390400302c1.run_wall_time_us, tablet-download-pool-d48de8e0c12b432ea5079390400302c1.run_wall_time_us) (tablet-download-pool-d9a1a818ab7e44a5ac5c67f42e1dfe3b.queue_time_us, tablet-download-pool-d9a1a818ab7e44a5ac5c67f42e1dfe3b.queue_time_us) (tablet-download-pool-d9a1a818ab7e44a5ac5c67f42e1dfe3b.run_cpu_time_us, tablet-download-pool-d9a1a818ab7e44a5ac5c67f42e1dfe3b.run_cpu_time_us) (tablet-download-pool-d9a1a818ab7e44a5ac5c67f42e1dfe3b.run_wall_time_us, tablet-download-pool-d9a1a818ab7e44a5ac5c67f42e1dfe3b.run_wall_time_us) (tablet-download-pool-ddf5ddac08364043bf2e27184d5d3213.queue_time_us, tablet-download-pool-ddf5ddac08364043bf2e27184d5d3213.queue_time_us) (tablet-download-pool-ddf5ddac08364043bf2e27184d5d3213.run_cpu_time_us, tablet-download-pool-ddf5ddac08364043bf2e27184d5d3213.run_cpu_time_us) (tablet-download-pool-ddf5ddac08364043bf2e27184d5d3213.run_wall_time_us, tablet-download-pool-ddf5ddac08364043bf2e27184d5d3213.run_wall_time_us) (tablet-download-pool-e483217b8761457d84edc49253877719.queue_time_us, tablet-download-pool-e483217b8761457d84edc49253877719.queue_time_us) (tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.queue_time_us, tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.queue_time_us) (tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.run_cpu_time_us, tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.run_cpu_time_us) (tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.run_wall_time_us, tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.run_wall_time_us) (tablet-download-pool-eef58fcca15041a39e685a7cdcb04f2b.queue_time_us, tablet-download-pool-eef58fcca15041a39e685a7cdcb04f2b.queue_time_us) (tablet-download-pool-eef58fcca15041a39e685a7cdcb04f2b.run_cpu_time_us, tablet-download-pool-eef58fcca15041a39e685a7cdcb04f2b.run_cpu_time_us) (tablet-download-pool-eef58fcca15041a39e685a7cdcb04f2b.run_wall_time_us, tablet-download-pool-eef58fcca15041a39e685a7cdcb04f2b.run_wall_time_us) (tablet-download-pool-f904b62401ff4528b3fa65dd4e11512a.queue_time_us, tablet-download-pool-f904b62401ff4528b3fa65dd4e11512a.queue_time_us) (tablet-download-pool-f904b62401ff4528b3fa65dd4e11512a.run_cpu_time_us, tablet-download-pool-f904b62401ff4528b3fa65dd4e11512a.run_cpu_time_us) (tablet-download-pool-f904b62401ff4528b3fa65dd4e11512a.run_wall_time_us, tablet-download-pool-f904b62401ff4528b3fa65dd4e11512a.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f20936e0dcd  google::LogMessage::Fail() at ??:0
    @     0x7f20936e4b93  google::LogMessage::SendToLog() at ??:0
I20240501 20:10:53.895110 20808 tablet_copy_client.cc:783] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:53.895334 20808 tablet_copy_client.cc:647] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
    @     0x7f20936e07cc  google::LogMessage::Flush() at ??:0
I20240501 20:10:53.896459 16263 consensus_queue.cc:237] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:53.897645 20808 tablet_copy_client.cc:523] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
    @     0x7f20936e1f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 20:10:53.900681 18188 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:53.901219 20702 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:53.901347 20808 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:53.903681 18188 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P 047275bda1074ac5a63325e7e1109c61 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
    @     0x7f2096421742  kudu::TraceMetrics::InternName() at ??:0
I20240501 20:10:53.907809 17928 raft_consensus.cc:1261] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:53.908313 20400 consensus_queue.cc:1035] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:53.911013 16003 raft_consensus.cc:1203] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
    @     0x7f20963edf8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240501 20:10:53.930130 16004 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:53.931681 17928 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
    @     0x7f20963e7cd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7f20a583a8de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f20a5852daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f20a597d480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7f20a59b5a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 20:10:53.973742 18190 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P 047275bda1074ac5a63325e7e1109c61 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:53.984671 18060 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
    @     0x7f20a5998981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 20:10:53.999053 15874 consensus_queue.cc:237] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:53.999373 20808 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:53.999155 20159 raft_consensus.cc:2932] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:53.999900 20808 tablet_bootstrap.cc:492] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:54.000514 20808 ts_tablet_manager.cc:1367] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.099s	user 0.058s	sys 0.026s
I20240501 20:10:54.001245 17795 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:54.001793 18059 raft_consensus.cc:1261] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:54.002261 20808 raft_consensus.cc:348] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:54.002504 20808 raft_consensus.cc:729] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:54.002465 20073 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:54.002691 20808 consensus_queue.cc:260] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:54.003034 20073 consensus_queue.cc:1035] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:54.003361 20808 ts_tablet_manager.cc:1398] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240501 20:10:54.004263 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-eef58fcca15041a39e685a7cdcb04f2b received from {username='slave'} at 127.8.68.199:53283
I20240501 20:10:54.004413 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session cad00548f33849e7a19cecd760e1e429-eef58fcca15041a39e685a7cdcb04f2b on tablet eef58fcca15041a39e685a7cdcb04f2b with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:54.005503 20749 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:54.006019 17795 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
    @     0x7f20a599870e  std::_Function_handler<>::_M_invoke() at ??:0
I20240501 20:10:54.010383 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f3eb13093b546d38fc71368f2083878 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240501 20:10:54.013187 15729 catalog_manager.cc:5484] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be reported cstate change: config changed from index 10 to 11, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:54.021631 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3eb13093b546d38fc71368f2083878 with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:54.031160 15729 catalog_manager.cc:5484] T 98eb95ded28d433aad9523586e29947f P d3917663631a4479b15355381486591c reported cstate change: config changed from index 8 to 9, VOTER 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:54.041586 16134 consensus_queue.cc:237] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:54.044665 15854 tablet_service.cc:1514] Processing DeleteTablet for tablet 98eb95ded28d433aad9523586e29947f with delete_type TABLET_DATA_TOMBSTONED (TS 51b81a5ae56b4dd99af1b0167c1da1e2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35708
W20240501 20:10:54.050980 16072 consensus_peers.cc:485] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58ae613944b941e694f790e02cfe5292. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:54.053973 20829 tablet_replica.cc:330] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:54.054162 20829 raft_consensus.cc:2229] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:54.054417 20829 raft_consensus.cc:2258] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:54.062732 20829 ts_tablet_manager.cc:1875] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:54.062846 18060 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:54.063551 20692 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:54.068966 20829 ts_tablet_manager.cc:1888] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 20:10:54.069072 20829 log.cc:1196] T 8f3eb13093b546d38fc71368f2083878 P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/8f3eb13093b546d38fc71368f2083878
I20240501 20:10:54.068665 18059 raft_consensus.cc:2932] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:54.069774 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet 8f3eb13093b546d38fc71368f2083878 (table rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]) successfully deleted
    @     0x7f20963f1b75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f20963f519c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240501 20:10:54.078344 16004 consensus_queue.cc:237] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } }
    @     0x7f20963f5019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240501 20:10:54.089665 20499 ts_tablet_manager.cc:902] T d48de8e0c12b432ea5079390400302c1 P d3917663631a4479b15355381486591c: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
    @     0x7f20963d001e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f20997556db  start_thread at ??:0
I20240501 20:10:54.101706 17795 raft_consensus.cc:1203] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
    @     0x7f209259371f  clone at ??:0
I20240501 20:10:54.130482 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98eb95ded28d433aad9523586e29947f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:54.140002 20833 tablet_replica.cc:330] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: stopping tablet replica
I20240501 20:10:54.140206 20833 raft_consensus.cc:2229] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:54.140384 20833 pending_rounds.cc:62] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: Trying to abort 1 pending ops.
I20240501 20:10:54.140434 20833 pending_rounds.cc:68] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022978063712051200 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "98eb95ded28d433aad9523586e29947f" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } } }
I20240501 20:10:54.140848 20833 raft_consensus.cc:2894] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 20:10:54.140950 20833 raft_consensus.cc:2866] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:54.141049 20833 raft_consensus.cc:2258] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:54.142175 20833 ts_tablet_manager.cc:1875] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:54.145200 20500 tablet_copy_client.cc:308] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:54.145886 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet 3643d6995910404fb8a7e6ef9c635f42 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:39585)
I20240501 20:10:54.146060 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet 3643d6995910404fb8a7e6ef9c635f42 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:39585: session id = cad00548f33849e7a19cecd760e1e429-3643d6995910404fb8a7e6ef9c635f42
I20240501 20:10:54.149013 16285 tablet_copy_source_session.cc:205] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:54.151052 19797 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:54.156278 18060 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:54.180255 16136 raft_consensus.cc:1261] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:54.180963 20361 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:54.185518 17929 raft_consensus.cc:1261] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 754fcd3b82f041568f546c6205f44272: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:54.186460 20531 consensus_queue.cc:1035] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:54.187233 20531 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:54.190589 16136 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:54.211230 15727 catalog_manager.cc:5484] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New cstate: current_term: 1 leader_uuid: "754fcd3b82f041568f546c6205f44272" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:54.222819 17926 raft_consensus.cc:2932] T 67d59d1654ef4ab3ac996f55a73b313f P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } }
I20240501 20:10:54.225477 20499 tablet_copy_client.cc:308] T d48de8e0c12b432ea5079390400302c1 P d3917663631a4479b15355381486591c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:54.226589 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet d48de8e0c12b432ea5079390400302c1 from peer d3917663631a4479b15355381486591c ({username='slave'} at 127.8.68.198:60703)
I20240501 20:10:54.226773 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet d48de8e0c12b432ea5079390400302c1 from peer d3917663631a4479b15355381486591c at {username='slave'} at 127.8.68.198:60703: session id = d3917663631a4479b15355381486591c-d48de8e0c12b432ea5079390400302c1
I20240501 20:10:54.229517 15895 tablet_copy_source_session.cc:205] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:54.234774 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:54.238948 20493 raft_consensus.cc:1050] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:54.239710 20493 consensus_queue.cc:237] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:54.242986 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48de8e0c12b432ea5079390400302c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.245019 20765 ts_tablet_manager.cc:902] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
I20240501 20:10:54.246181 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:54.270246 20500 ts_tablet_manager.cc:1875] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:10:54.273479 20765 tablet_copy_client.cc:308] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:54.273720 18058 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:54.274098 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:49547)
I20240501 20:10:54.274240 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:49547: session id = 9a0f0037700246bdb65e05aef95fb2be-d9a1a818ab7e44a5ac5c67f42e1dfe3b
I20240501 20:10:54.274210 20073 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:54.275543 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:54.277346 15895 tablet_copy_source_session.cc:205] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:54.278360 20765 ts_tablet_manager.cc:1875] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:10:54.288331 16136 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:54.288846 20073 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:54.290552 20853 ts_tablet_manager.cc:902] T d3ba52a871714255aa1bd34d9daca920 P d3917663631a4479b15355381486591c: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:54.290678 16263 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
F20240501 20:10:54.290969 20853 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.queue_time_us, tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.queue_time_us) (tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.run_cpu_time_us, tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.run_cpu_time_us) (tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.run_wall_time_us, tablet-download-pool-2e6b15e4aebf4b1f9a3176f87083703e.run_wall_time_us) (tablet-download-pool-2fce37f2e8f74a119b8bd0c34b1a35ac.queue_time_us, tablet-download-pool-2fce37f2e8f74a119b8bd0c34b1a35ac.queue_time_us) (tablet-download-pool-2fce37f2e8f74a119b8bd0c34b1a35ac.run_cpu_time_us, tablet-download-pool-2fce37f2e8f74a119b8bd0c34b1a35ac.run_cpu_time_us) (tablet-download-pool-2fce37f2e8f74a119b8bd0c34b1a35ac.run_wall_time_us, tablet-download-pool-2fce37f2e8f74a119b8bd0c34b1a35ac.run_wall_time_us) (tablet-download-pool-3643d6995910404fb8a7e6ef9c635f42.queue_time_us, tablet-download-pool-3643d6995910404fb8a7e6ef9c635f42.queue_time_us) (tablet-download-pool-3643d6995910404fb8a7e6ef9c635f42.run_cpu_time_us, tablet-download-pool-3643d6995910404fb8a7e6ef9c635f42.run_cpu_time_us) (tablet-download-pool-3643d6995910404fb8a7e6ef9c635f42.run_wall_time_us, tablet-download-pool-3643d6995910404fb8a7e6ef9c635f42.run_wall_time_us) (tablet-download-pool-67d59d1654ef4ab3ac996f55a73b313f.queue_time_us, tablet-download-pool-67d59d1654ef4ab3ac996f55a73b313f.queue_time_us) (tablet-download-pool-67d59d1654ef4ab3ac996f55a73b313f.run_cpu_time_us, tablet-download-pool-67d59d1654ef4ab3ac996f55a73b313f.run_cpu_time_us) (tablet-download-pool-67d59d1654ef4ab3ac996f55a73b313f.run_wall_time_us, tablet-download-pool-67d59d1654ef4ab3ac996f55a73b313f.run_wall_time_us) (tablet-download-pool-98eb95ded28d433aad9523586e29947f.queue_time_us, tablet-download-pool-98eb95ded28d433aad9523586e29947f.queue_time_us) (tablet-download-pool-98eb95ded28d433aad9523586e29947f.run_cpu_time_us, tablet-download-pool-98eb95ded28d433aad9523586e29947f.run_cpu_time_us) (tablet-download-pool-98eb95ded28d433aad9523586e29947f.run_wall_time_us, tablet-download-pool-98eb95ded28d433aad9523586e29947f.run_wall_time_us) (tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.queue_time_us, tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.queue_time_us) (tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.run_cpu_time_us, tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.run_cpu_time_us) (tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.run_wall_time_us, tablet-download-pool-b016fc1d28524a02bd762b74c6b08375.run_wall_time_us) (tablet-download-pool-c538581c71f646b5bc8d71cb6981f201.queue_time_us, tablet-download-pool-c538581c71f646b5bc8d71cb6981f201.queue_time_us) (tablet-download-pool-c538581c71f646b5bc8d71cb6981f201.run_cpu_time_us, tablet-download-pool-c538581c71f646b5bc8d71cb6981f201.run_cpu_time_us) (tablet-download-pool-c538581c71f646b5bc8d71cb6981f201.run_wall_time_us, tablet-download-pool-c538581c71f646b5bc8d71cb6981f201.run_wall_time_us) (tablet-download-pool-c74a6da428b04df58b7edcdf88d81cca.queue_time_us, tablet-download-pool-c74a6da428b04df58b7edcdf88d81cca.queue_time_us) (tablet-download-pool-c74a6da428b04df58b7edcdf88d81cca.run_cpu_time_us, tablet-download-pool-c74a6da428b04df58b7edcdf88d81cca.run_cpu_time_us) (tablet-download-pool-c74a6da428b04df58b7edcdf88d81cca.run_wall_time_us, tablet-download-pool-c74a6da428b04df58b7edcdf88d81cca.run_wall_time_us) (tablet-download-pool-d3ba52a871714255aa1bd34d9daca920.queue_time_us, tablet-download-pool-d3ba52a871714255aa1bd34d9daca920.queue_time_us) (tablet-download-pool-d48de8e0c12b432ea5079390400302c1.queue_time_us, tablet-download-pool-d48de8e0c12b432ea5079390400302c1.queue_time_us) (tablet-download-pool-d48de8e0c12b432ea5079390400302c1.run_cpu_time_us, tablet-download-pool-d48de8e0c12b432ea5079390400302c1.run_cpu_time_us) (tablet-download-pool-d48de8e0c12b432ea5079390400302c1.run_wall_time_us, tablet-download-pool-d48de8e0c12b432ea5079390400302c1.run_wall_time_us) (tablet-download-pool-e483217b8761457d84edc49253877719.queue_time_us, tablet-download-pool-e483217b8761457d84edc49253877719.queue_time_us) (tablet-download-pool-e483217b8761457d84edc49253877719.run_cpu_time_us, tablet-download-pool-e483217b8761457d84edc49253877719.run_cpu_time_us) (tablet-download-pool-e483217b8761457d84edc49253877719.run_wall_time_us, tablet-download-pool-e483217b8761457d84edc49253877719.run_wall_time_us) (tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.queue_time_us, tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.queue_time_us) (tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.run_cpu_time_us, tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.run_cpu_time_us) (tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.run_wall_time_us, tablet-download-pool-eb90224d08f84a2faf7dcc063c6af56e.run_wall_time_us) (tablet-download-pool-f345303c72774848bb60e806cff22d12.queue_time_us, tablet-download-pool-f345303c72774848bb60e806cff22d12.queue_time_us) (tablet-download-pool-f345303c72774848bb60e806cff22d12.run_cpu_time_us, tablet-download-pool-f345303c72774848bb60e806cff22d12.run_cpu_time_us) (tablet-download-pool-f345303c72774848bb60e806cff22d12.run_wall_time_us, tablet-download-pool-f345303c72774848bb60e806cff22d12.run_wall_time_us) (tablet-download-pool-f3761f02c20a4448acd89f4a37f1193c.queue_time_us, tablet-download-pool-f3761f02c20a4448acd89f4a37f1193c.queue_time_us) (tablet-download-pool-f3761f02c20a4448acd89f4a37f1193c.run_cpu_time_us, tablet-download-pool-f3761f02c20a4448acd89f4a37f1193c.run_cpu_time_us) (tablet-download-pool-f3761f02c20a4448acd89f4a37f1193c.run_wall_time_us, tablet-download-pool-f3761f02c20a4448acd89f4a37f1193c.run_wall_time_us) (tablet-download-pool-f7fb9bc4cd73492da44bde1eecd627dc.queue_time_us, tablet-download-pool-f7fb9bc4cd73492da44bde1eecd627dc.queue_time_us) (tablet-download-pool-f7fb9bc4cd73492da44bde1eecd627dc.run_cpu_time_us, tablet-download-pool-f7fb9bc4cd73492da44bde1eecd627dc.run_cpu_time_us) (tablet-download-pool-f7fb9bc4cd73492da44bde1eecd627dc.run_wall_time_us, tablet-download-pool-f7fb9bc4cd73492da44bde1eecd627dc.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240501 20:10:54.291245 20692 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
*** Check failure stack trace: ***
    @     0x7f0b2a6c2dcd  google::LogMessage::Fail() at ??:0
    @     0x7f0b2a6c6b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f0b2a6c27cc  google::LogMessage::Flush() at ??:0
    @     0x7f0b2a6c3f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f0b2d403742  kudu::TraceMetrics::InternName() at ??:0
I20240501 20:10:54.310766 16263 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:54.311694 20073 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:54.316103 18058 raft_consensus.cc:1203] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 20:10:54.316047 18057 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
    @     0x7f0b2d3cff8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240501 20:10:54.321563 15727 catalog_manager.cc:5484] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:54.329072 17928 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
    @     0x7f0b2d3c9cd6  kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 20:10:54.331806 20749 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:54.332743 15728 catalog_manager.cc:5484] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 6 to 7, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:54.379426 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:54.395643 20500 ts_tablet_manager.cc:1888] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20240501 20:10:54.424638 20500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3643d6995910404fb8a7e6ef9c635f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.428131 20500 tablet_copy_client.cc:783] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:54.428416 20500 tablet_copy_client.cc:647] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:54.437129 20500 tablet_copy_client.cc:523] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:54.453927 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:54.466025 20500 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:54.718511 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:54.741583 20500 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:54.742048 20500 tablet_bootstrap.cc:492] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:54.742419 20500 ts_tablet_manager.cc:1367] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.276s	user 0.057s	sys 0.010s
I20240501 20:10:54.743333 20500 raft_consensus.cc:348] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:54.743655 20500 raft_consensus.cc:729] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:54.743906 20500 consensus_queue.cc:260] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:54.334182 15873 consensus_queue.cc:237] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:54.337774 20749 raft_consensus.cc:1050] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER 754fcd3b82f041568f546c6205f44272 to VOTER
I20240501 20:10:54.395891 20833 ts_tablet_manager.cc:1888] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:54.749678 20833 log.cc:1196] T 98eb95ded28d433aad9523586e29947f P 51b81a5ae56b4dd99af1b0167c1da1e2: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/98eb95ded28d433aad9523586e29947f
W20240501 20:10:54.676844 15809 connection.cc:535] client connection to 127.8.68.200:38791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:54.750522 15809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:54.750401 20749 consensus_queue.cc:237] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: false } }
I20240501 20:10:54.765219 20500 ts_tablet_manager.cc:1398] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.023s	user 0.000s	sys 0.002s
W20240501 20:10:54.766651 15811 consensus_peers.cc:485] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer cad00548f33849e7a19cecd760e1e429 (127.8.68.199:44275): Couldn't send request to peer cad00548f33849e7a19cecd760e1e429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 627e9e5c120747f1916855ea83008657. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:54.778545 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:54.775760 15719 catalog_manager.cc:4830] TS 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): tablet 98eb95ded28d433aad9523586e29947f (table rebalance_test_table_3 [id=f219e025e794415a83c70d7e68f6b884]) successfully deleted
W20240501 20:10:54.782918 15809 consensus_peers.cc:485] T 2e6b15e4aebf4b1f9a3176f87083703e P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:54.794042 15809 consensus_peers.cc:485] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:54.794147 15809 consensus_peers.cc:485] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:54.794232 15809 consensus_peers.cc:462] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:54.799896 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:10:54.810748 15809 consensus_peers.cc:485] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:54.810928 15809 consensus_peers.cc:485] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:54.362722 15986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:10:54.677855 15940 connection.cc:535] client connection to 127.8.68.200:38791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:54.700608 15985 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:54.336535 20820 raft_consensus.cc:1050] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
W20240501 20:10:54.898690 15940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:54.764225 16004 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:54.775147 16003 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:54.899500 20820 consensus_queue.cc:237] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
I20240501 20:10:54.900039 20493 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:54.900669 20493 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:54.430985 16246 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:54.435427 16245 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:10:54.676374 16200 connection.cc:535] client connection to 127.8.68.200:38791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:54.769896 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-3643d6995910404fb8a7e6ef9c635f42 received from {username='slave'} at 127.8.68.199:39585
I20240501 20:10:54.902508 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session cad00548f33849e7a19cecd760e1e429-3643d6995910404fb8a7e6ef9c635f42 on tablet 3643d6995910404fb8a7e6ef9c635f42 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:54.771234 16244 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:54.793727 20400 raft_consensus.cc:1050] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER d3917663631a4479b15355381486591c to VOTER
I20240501 20:10:54.407209 20765 ts_tablet_manager.cc:1888] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:10:54.971211 20765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.929270 20073 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
W20240501 20:10:54.971186 20400 raft_consensus.cc:1054] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e: Unable to promote non-voter d3917663631a4479b15355381486591c: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }.
  Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } }
W20240501 20:10:54.904790 16200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:54.323752 16136 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:54.412230 17774 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:54.659380 20922 ts_tablet_manager.cc:902] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553)
I20240501 20:10:54.707026 17773 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:54.765543 17795 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:54.777598 17792 raft_consensus.cc:1261] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:54.355693 17928 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
    @     0x7f0b3c81c8de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 20:10:54.406939 20499 tablet_copy_client.cc:783] T d48de8e0c12b432ea5079390400302c1 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:54.979240 20499 tablet_copy_client.cc:647] T d48de8e0c12b432ea5079390400302c1 P d3917663631a4479b15355381486591c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:54.979470 20952 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:54.980055 20952 consensus_queue.cc:1035] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:54.549401 19797 raft_consensus.cc:1050] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:54.983273 19797 consensus_queue.cc:237] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
W20240501 20:10:54.675797 16070 connection.cc:535] client connection to 127.8.68.200:38791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:54.984980 16070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:54.985635 15728 catalog_manager.cc:5484] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 7 to 8, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:54.918687 17929 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P d3917663631a4479b15355381486591c [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:54.987872 20400 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:54.990695 20499 tablet_copy_client.cc:523] T d48de8e0c12b432ea5079390400302c1 P d3917663631a4479b15355381486591c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:54.994652 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:54.996975 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:54.997321 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
    @     0x7f0b3c834daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 20:10:55.007300 20765 tablet_copy_client.cc:783] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:55.007540 20765 tablet_copy_client.cc:647] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:54.994264 20499 tablet_bootstrap.cc:492] T d48de8e0c12b432ea5079390400302c1 P d3917663631a4479b15355381486591c: Bootstrap starting.
    @     0x7f0b3c95f480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7f0b3c997a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 20:10:55.054373 17927 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P d3917663631a4479b15355381486591c [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:55.063632 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:55.080680 16200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:38791, user_credentials={real_user=slave}} blocked reactor thread for 178252us
W20240501 20:10:55.158973 16200 consensus_peers.cc:485] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
    @     0x7f0b3c97a981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 20:10:55.110435 17908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:55.011623 17792 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
I20240501 20:10:55.187186 18060 raft_consensus.cc:1203] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
I20240501 20:10:55.111258 17906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:55.016189 20765 tablet_copy_client.cc:523] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:55.050774 17794 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:55.079257 17791 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 754fcd3b82f041568f546c6205f44272 (127.8.68.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: false } } }
I20240501 20:10:55.080228 20922 tablet_copy_client.cc:308] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:46553
I20240501 20:10:55.191900 20765 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:55.111295 17907 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:55.112659 17905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:10:55.204116 16200 consensus_peers.cc:485] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
    @     0x7f0b3c97a70e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f0b2d3d3b75  kudu::ThreadPool::DispatchThread() at ??:0
I20240501 20:10:55.217860 17773 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:55.057765 20834 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:54.929926 16003 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER cad00548f33849e7a19cecd760e1e429 (127.8.68.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } } }
W20240501 20:10:55.079349 15939 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.198:44869, user_credentials={real_user=slave}} blocked reactor thread for 94945.3us
I20240501 20:10:55.274967 19797 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:10:54.977170 15940 consensus_peers.cc:485] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.276947 15940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:38791, user_credentials={real_user=slave}} blocked reactor thread for 299747us
W20240501 20:10:55.277107 15940 consensus_peers.cc:485] T 0f25f3a86f0b494b89fe8f86aae4f4cd P 754fcd3b82f041568f546c6205f44272 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.277215 15940 consensus_peers.cc:485] T f904b62401ff4528b3fa65dd4e11512a P 754fcd3b82f041568f546c6205f44272 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:55.078574 20073 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 754fcd3b82f041568f546c6205f44272 (127.8.68.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: false } } }
I20240501 20:10:55.213798 16244 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:55.194500 15727 catalog_manager.cc:5484] T ddf5ddac08364043bf2e27184d5d3213 P 9a0f0037700246bdb65e05aef95fb2be reported cstate change: config changed from index 10 to 11, 754fcd3b82f041568f546c6205f44272 (127.8.68.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: false } } }
W20240501 20:10:55.063963 16070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.200:38791, user_credentials={real_user=slave}} blocked reactor thread for 79032.2us
W20240501 20:10:55.128186 15809 consensus_peers.cc:462] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:55.133240 15873 raft_consensus.cc:1261] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:10:55.283666 15809 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.8.68.200:38791, user_credentials={real_user=slave}} blocked reactor thread for 155470us
I20240501 20:10:55.289237 20500 ts_tablet_manager.cc:902] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: Initiating tablet copy from peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513)
    @     0x7f0b2d3d719c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240501 20:10:55.326328 20400 consensus_queue.cc:1035] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
    @     0x7f0b2d3d7019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240501 20:10:55.348698 16263 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:55.356492 19797 raft_consensus.cc:1050] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:55.356977 16156 tablet_copy_service.cc:140] P 042aee20cf4e4e63be3ecc013fda59c9: Received BeginTabletCopySession request for tablet 58ae613944b941e694f790e02cfe5292 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:53423)
I20240501 20:10:55.357142 16156 tablet_copy_service.cc:161] P 042aee20cf4e4e63be3ecc013fda59c9: Beginning new tablet copy session on tablet 58ae613944b941e694f790e02cfe5292 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:53423: session id = 9a0f0037700246bdb65e05aef95fb2be-58ae613944b941e694f790e02cfe5292
I20240501 20:10:55.357318 19797 consensus_queue.cc:237] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:55.359999 16156 tablet_copy_source_session.cc:205] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 20:10:55.360967 16070 consensus_peers.cc:485] T 5682259bc54c469e9c6a08723c1bbee0 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.361132 16070 consensus_peers.cc:485] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:55.363158 20922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ae613944b941e694f790e02cfe5292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.364724 20765 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:55.365206 20765 tablet_bootstrap.cc:492] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:55.365468 20765 ts_tablet_manager.cc:1367] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.174s	user 0.048s	sys 0.005s
    @     0x7f0b2d3b201e  kudu::Thread::SuperviseThread() at ??:0
I20240501 20:10:55.366329 20765 raft_consensus.cc:348] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:55.366621 20765 raft_consensus.cc:729] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:55.366881 20765 consensus_queue.cc:260] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:55.367511 20765 ts_tablet_manager.cc:1398] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.003s
    @     0x7f0b307376db  start_thread at ??:0
    @     0x7f0b2957571f  clone at ??:0
I20240501 20:10:55.375221 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-d9a1a818ab7e44a5ac5c67f42e1dfe3b received from {username='slave'} at 127.8.68.197:49547
I20240501 20:10:55.375375 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-d9a1a818ab7e44a5ac5c67f42e1dfe3b on tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:55.378782 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:10:55.394807 16202 consensus_peers.cc:485] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e -> Peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Couldn't send request to peer 9a0f0037700246bdb65e05aef95fb2be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c538581c71f646b5bc8d71cb6981f201. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:55.396600 18060 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:55.396880 20400 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:55.399544 20820 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:55.401157 15874 raft_consensus.cc:2932] T eb90224d08f84a2faf7dcc063c6af56e P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } }
I20240501 20:10:55.406764 16003 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:55.407303 15728 catalog_manager.cc:5484] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 7 to 8, d3917663631a4479b15355381486591c (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:55.410099 20692 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:55.413920 18058 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:55.414594 20834 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:55.421674 16004 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:55.423053 16005 raft_consensus.cc:2932] T ddf5ddac08364043bf2e27184d5d3213 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 754fcd3b82f041568f546c6205f44272 (127.8.68.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { promote: false } } }
I20240501 20:10:55.428704 20922 tablet_copy_client.cc:783] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:55.428930 20922 tablet_copy_client.cc:647] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:55.434752 20500 tablet_copy_client.cc:308] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:35513
I20240501 20:10:55.435379 15895 tablet_copy_service.cc:140] P 51b81a5ae56b4dd99af1b0167c1da1e2: Received BeginTabletCopySession request for tablet 627e9e5c120747f1916855ea83008657 from peer cad00548f33849e7a19cecd760e1e429 ({username='slave'} at 127.8.68.199:37603)
I20240501 20:10:55.435549 15895 tablet_copy_service.cc:161] P 51b81a5ae56b4dd99af1b0167c1da1e2: Beginning new tablet copy session on tablet 627e9e5c120747f1916855ea83008657 from peer cad00548f33849e7a19cecd760e1e429 at {username='slave'} at 127.8.68.199:37603: session id = cad00548f33849e7a19cecd760e1e429-627e9e5c120747f1916855ea83008657
I20240501 20:10:55.438036 15895 tablet_copy_source_session.cc:205] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:55.439662 20500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 627e9e5c120747f1916855ea83008657. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.447350 15985 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:55.462311 16005 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:55.457823 15873 consensus_queue.cc:237] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:55.455814 16263 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:55.453946 20500 tablet_copy_client.cc:783] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:55.490635 20500 tablet_copy_client.cc:647] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: tablet copy: Starting download of 1 WAL segments...
W20240501 20:10:55.448454 16070 consensus_peers.cc:485] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:55.439208 20922 tablet_copy_client.cc:523] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 20:10:55.454747 21015 log.cc:924] Time spent T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20240501 20:10:55.492643 17793 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:55.485441 20400 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:55.458460 20692 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:55.459331 20834 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:55.498756 15728 catalog_manager.cc:5484] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:55.499715 20922 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
I20240501 20:10:55.503770 20500 tablet_copy_client.cc:523] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:55.485878 18058 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } } }
I20240501 20:10:55.506145 21008 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:55.508980 15729 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 9 to 10, NON_VOTER 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:55.513311 20500 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: Bootstrap starting.
I20240501 20:10:55.514801 16136 consensus_queue.cc:237] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:55.517036 16005 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:55.517575 20692 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:55.525673 19797 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:55.531428 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f345303c72774848bb60e806cff22d12 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:55.532829 15731 catalog_manager.cc:5484] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 8 to 9, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:55.535068 18058 raft_consensus.cc:1261] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:55.594947 16263 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.544713 20692 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.542047 16005 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:55.547588 16003 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.660050 15731 catalog_manager.cc:5484] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:55.535632 21019 consensus_queue.cc:1035] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:10:55.541309 15809 consensus_peers.cc:462] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
W20240501 20:10:55.661312 15809 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.8.68.200:38791, user_credentials={real_user=slave}} blocked reactor thread for 119969us
I20240501 20:10:55.670148 16136 consensus_queue.cc:237] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:55.675055 16265 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:55.675698 21008 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:55.679106 16005 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:55.679695 21008 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:55.685447 16003 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.683058 21019 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.706670 15729 catalog_manager.cc:5484] T eef58fcca15041a39e685a7cdcb04f2b P 754fcd3b82f041568f546c6205f44272 reported cstate change: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.710915 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c74a6da428b04df58b7edcdf88d81cca with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 20:10:55.713749 15719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
W20240501 20:10:55.720417 15719 catalog_manager.cc:4573] TS 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eef58fcca15041a39e685a7cdcb04f2b: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:55.723578 15731 catalog_manager.cc:5484] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:55.727313 20820 raft_consensus.cc:1050] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:55.646931 20500 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:55.749573 20500 tablet_bootstrap.cc:492] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: Bootstrap complete.
I20240501 20:10:55.749869 20500 ts_tablet_manager.cc:1367] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: Time spent bootstrapping tablet: real 0.237s	user 0.050s	sys 0.001s
I20240501 20:10:55.750802 20500 raft_consensus.cc:348] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:55.751154 20500 raft_consensus.cc:729] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cad00548f33849e7a19cecd760e1e429, State: Initialized, Role: LEARNER
I20240501 20:10:55.751428 20500 consensus_queue.cc:260] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: NON_VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: true } }
I20240501 20:10:55.752120 20500 ts_tablet_manager.cc:1398] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:10:55.728241 20820 consensus_queue.cc:237] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:55.538339 17792 raft_consensus.cc:1261] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:55.646083 20922 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:55.682111 21008 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.753975 20922 tablet_bootstrap.cc:492] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:55.754272 20922 ts_tablet_manager.cc:1367] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.260s	user 0.041s	sys 0.008s
I20240501 20:10:55.755118 20922 raft_consensus.cc:348] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:55.755385 20922 raft_consensus.cc:729] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:55.755611 20922 consensus_queue.cc:260] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:55.756187 20922 ts_tablet_manager.cc:1398] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:55.687108 17793 raft_consensus.cc:1203] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240501 20:10:55.655095 18058 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.757939 15720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eef58fcca15041a39e685a7cdcb04f2b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:55.758731 16156 tablet_copy_service.cc:342] P 042aee20cf4e4e63be3ecc013fda59c9: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-58ae613944b941e694f790e02cfe5292 received from {username='slave'} at 127.8.68.197:53423
I20240501 20:10:55.758843 16156 tablet_copy_service.cc:433] P 042aee20cf4e4e63be3ecc013fda59c9: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-58ae613944b941e694f790e02cfe5292 on tablet 58ae613944b941e694f790e02cfe5292 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:55.729842 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet c74a6da428b04df58b7edcdf88d81cca with delete_type TABLET_DATA_TOMBSTONED (TS 042aee20cf4e4e63be3ecc013fda59c9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55668
I20240501 20:10:55.684854 18060 raft_consensus.cc:2932] T c74a6da428b04df58b7edcdf88d81cca P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.769029 18059 raft_consensus.cc:1261] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:10:55.770817 16069 connection.cc:535] client connection to 127.8.68.198:44869 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:55.772639 15939 connection.cc:535] client connection to 127.8.68.198:44869 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:55.773221 16199 connection.cc:535] client connection to 127.8.68.198:44869 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:55.774251 15895 tablet_copy_service.cc:342] P 51b81a5ae56b4dd99af1b0167c1da1e2: Request end of tablet copy session cad00548f33849e7a19cecd760e1e429-627e9e5c120747f1916855ea83008657 received from {username='slave'} at 127.8.68.199:37603
I20240501 20:10:55.774574 15895 tablet_copy_service.cc:433] P 51b81a5ae56b4dd99af1b0167c1da1e2: ending tablet copy session cad00548f33849e7a19cecd760e1e429-627e9e5c120747f1916855ea83008657 on tablet 627e9e5c120747f1916855ea83008657 with peer cad00548f33849e7a19cecd760e1e429
I20240501 20:10:55.775979 21030 consensus_queue.cc:1035] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:55.776575 21030 consensus_queue.cc:1035] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:55.776298 16265 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.778725 17792 raft_consensus.cc:2932] T f345303c72774848bb60e806cff22d12 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:55.779773 18058 raft_consensus.cc:2932] T eef58fcca15041a39e685a7cdcb04f2b P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 047275bda1074ac5a63325e7e1109c61 (127.8.68.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
W20240501 20:10:55.782967 15808 connection.cc:535] client connection to 127.8.68.198:44869 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:55.796552 21018 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:55.771544 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.198:44869 (local address 127.8.68.195:36737)
0501 20:10:55.771775 (+   231us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:55.771792 (+    17us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:55.771910 (+   118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:55.795616 (+ 23706us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127}
W20240501 20:10:55.798053 20975 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:55.773421 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.198:44869 (local address 127.8.68.194:49863)
0501 20:10:55.773705 (+   284us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:55.773721 (+    16us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:55.773837 (+   116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:55.797245 (+ 23408us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":182}
W20240501 20:10:55.798259 15939 consensus_peers.cc:485] T f7fb9bc4cd73492da44bde1eecd627dc P 754fcd3b82f041568f546c6205f44272 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.798410 15939 consensus_peers.cc:485] T f3761f02c20a4448acd89f4a37f1193c P 754fcd3b82f041568f546c6205f44272 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.798507 15939 consensus_peers.cc:485] T 67d59d1654ef4ab3ac996f55a73b313f P 754fcd3b82f041568f546c6205f44272 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.799666 20999 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:55.775173 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.198:44869 (local address 127.8.68.196:37213)
0501 20:10:55.775467 (+   294us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:55.775482 (+    15us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:55.775597 (+   115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:55.798941 (+ 23344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":201}
W20240501 20:10:55.799923 16199 consensus_peers.cc:462] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:55.800052 16199 consensus_peers.cc:485] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.800133 16199 consensus_peers.cc:485] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.800280 16199 consensus_peers.cc:485] T b016fc1d28524a02bd762b74c6b08375 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.800359 16199 consensus_peers.cc:485] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.801499 20953 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:55.785161 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.198:44869 (local address 127.8.68.193:52425)
0501 20:10:55.785476 (+   315us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:55.785493 (+    17us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:55.785612 (+   119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:55.800802 (+ 15190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":214}
W20240501 20:10:55.801959 15808 consensus_peers.cc:485] T 2fce37f2e8f74a119b8bd0c34b1a35ac P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.802125 15808 consensus_peers.cc:485] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:55.802240 15808 consensus_peers.cc:485] T c74a6da428b04df58b7edcdf88d81cca P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:55.816620 21071 tablet_replica.cc:330] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9: stopping tablet replica
I20240501 20:10:55.816865 21071 raft_consensus.cc:2229] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:55.817137 21071 raft_consensus.cc:2258] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:55.818136 21071 ts_tablet_manager.cc:1875] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:10:55.830739 15718 connection.cc:535] client connection to 127.8.68.198:44869 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:55.837477 16199 consensus_peers.cc:485] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:55.838606 18060 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:10:55.838677 17793 raft_consensus.cc:1261] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:10:55.839198 20159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:10:55.839758 20159 consensus_queue.cc:1035] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:10:55.865501 20159 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.871194 18058 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.872458 17793 raft_consensus.cc:2932] T 3643d6995910404fb8a7e6ef9c635f42 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:55.873931 15731 catalog_manager.cc:5484] T 3643d6995910404fb8a7e6ef9c635f42 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 13 to 14, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:55.881731 18058 raft_consensus.cc:1203] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:10:55.902704 21071 ts_tablet_manager.cc:1888] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:55.914391 21071 log.cc:1196] T c74a6da428b04df58b7edcdf88d81cca P 042aee20cf4e4e63be3ecc013fda59c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/c74a6da428b04df58b7edcdf88d81cca
I20240501 20:10:55.915274 15720 catalog_manager.cc:4830] TS 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): tablet c74a6da428b04df58b7edcdf88d81cca (table rebalance_test_table_0 [id=d1335198d9bd43dcbdaeb39d406d227b]) successfully deleted
I20240501 20:10:55.981499 20922 ts_tablet_manager.cc:902] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: Initiating tablet copy from peer f856e0d1074b431eb19763847958b89e (127.8.68.196:36919)
I20240501 20:10:56.039745 20922 tablet_copy_client.cc:308] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.196:36919
I20240501 20:10:56.040527 16285 tablet_copy_service.cc:140] P f856e0d1074b431eb19763847958b89e: Received BeginTabletCopySession request for tablet c538581c71f646b5bc8d71cb6981f201 from peer 9a0f0037700246bdb65e05aef95fb2be ({username='slave'} at 127.8.68.197:55615)
I20240501 20:10:56.040701 16285 tablet_copy_service.cc:161] P f856e0d1074b431eb19763847958b89e: Beginning new tablet copy session on tablet c538581c71f646b5bc8d71cb6981f201 from peer 9a0f0037700246bdb65e05aef95fb2be at {username='slave'} at 127.8.68.197:55615: session id = 9a0f0037700246bdb65e05aef95fb2be-c538581c71f646b5bc8d71cb6981f201
I20240501 20:10:56.043426 16285 tablet_copy_source_session.cc:205] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:56.044845 20922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c538581c71f646b5bc8d71cb6981f201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.048461 18238 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 20:10:37.658268 18245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:37.658627 18245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:37.670229 18245 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:41.025580 18245 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:10:42.782344 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.782507 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.810886 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:42.811300 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.811383 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:42.811478 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:42.811568 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.811702 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.811774 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.811924 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server d3917663631a4479b15355381486591c
W20240501 20:10:42.812005 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.812094 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.812166 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:42.812276 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:43.367976 18245 rebalancer_tool.cc:1084] tablet 2fce37f2e8f74a119b8bd0c34b1a35ac: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move ignored: Illegal state: tablet 2fce37f2e8f74a119b8bd0c34b1a35ac: replica 047275bda1074ac5a63325e7e1109c61 is already present
I20240501 20:10:43.368194 18245 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:46.395591 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:46.395763 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:46.395869 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:46.396066 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server d3917663631a4479b15355381486591c
W20240501 20:10:46.396178 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server d3917663631a4479b15355381486591c
W20240501 20:10:46.396411 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:46.396550 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:46.396680 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:46.396780 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
W20240501 20:10:46.396912 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server d3917663631a4479b15355381486591c
W20240501 20:10:46.397022 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:46.397243 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:46.931850 18245 rebalancer_tool.cc:1074] tablet 2e6b15e4aebf4b1f9a3176f87083703e: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:47.244863 18245 rebalancer_tool.cc:1074] tablet f638f341ff2341dc82889334eb87e3d0: '042aee20cf4e4e63be3ecc013fda59c9' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:47.418654 18245 rebalancer_tool.cc:1074] tablet ddf5ddac08364043bf2e27184d5d3213: '754fcd3b82f041568f546c6205f44272' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:47.794420 18245 rebalancer_tool.cc:1084] tablet e483217b8761457d84edc49253877719: 'f856e0d1074b431eb19763847958b89e' -> '047275bda1074ac5a63325e7e1109c61' move ignored: Not found: Peer with uuid f856e0d1074b431eb19763847958b89e not found in consensus config
I20240501 20:10:47.794560 18245 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:49.042821 20006 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.8.68.254:46529, user_credentials={real_user=slave}} blocked reactor thread for 99552.7us
W20240501 20:10:49.785235 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 042aee20cf4e4e63be3ecc013fda59c9
W20240501 20:10:49.785399 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:49.857467 18245 rebalancer_tool.cc:1084] tablet 5682259bc54c469e9c6a08723c1bbee0: 'f856e0d1074b431eb19763847958b89e' -> '047275bda1074ac5a63325e7e1109c61' move ignored: Illegal state: tablet 5682259bc54c469e9c6a08723c1bbee0: replica 047275bda1074ac5a63325e7e1109c61 is already present
I20240501 20:10:49.857601 18245 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:50.368382 18245 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.510s	user 0.047s	sys 0.115s
W20240501 20:10:50.368592 18245 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.510s	user 0.047s	sys 0.116s
W20240501 20:10:52.311226 18245 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 042aee20cf4e4e63be3ecc013fda59c9
I20240501 20:10:52.658941 18245 rebalancer_tool.cc:1074] tablet eef58fcca15041a39e685a7cdcb04f2b: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:52.846443 18245 rebalancer_tool.cc:1084] tablet 98eb95ded28d433aad9523586e29947f: 'f856e0d1074b431eb19763847958b89e' -> 'd3917663631a4479b15355381486591c' move ignored: Illegal state: tablet 98eb95ded28d433aad9523586e29947f: replica d3917663631a4479b15355381486591c is already present
I20240501 20:10:52.846590 18245 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:54.693014 20882 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:54.263681 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.200:38791 (local address 127.0.0.1:40626)
0501 20:10:54.304952 (+ 41271us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:54.304971 (+    19us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:54.305084 (+   113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:54.691748 (+386664us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41173,"thread_start_us":46230,"threads_started":1}
Illegal state: tablet server 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): unacceptable health status UNAVAILABLE

I20240501 20:10:56.064221 20922 tablet_copy_client.cc:783] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:56.064673 20922 tablet_copy_client.cc:647] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:56.066919 18242 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 20:10:37.565454 18244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:37.565861 18244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:37.577910 18244 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:40.114002 18244 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:10:41.728142 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9397d415e86d427d8fe2f9e906b59e2c: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.728632 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9397d415e86d427d8fe2f9e906b59e2c: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:41.728742 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.728849 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server d3917663631a4479b15355381486591c
W20240501 20:10:41.728965 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:41.729084 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.729387 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.729506 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:41.729621 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server d3917663631a4479b15355381486591c
W20240501 20:10:41.729712 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:41.729795 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.729893 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:41.729983 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:41.730108 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server d3917663631a4479b15355381486591c
W20240501 20:10:41.730233 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:41.827890 18244 rebalancer_tool.cc:1084] tablet 2fce37f2e8f74a119b8bd0c34b1a35ac: '754fcd3b82f041568f546c6205f44272' -> 'd3917663631a4479b15355381486591c' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:41.828140 18244 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:44.186167 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9397d415e86d427d8fe2f9e906b59e2c: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:44.188030 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:44.188374 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:44.188525 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:44.190563 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server d3917663631a4479b15355381486591c
W20240501 20:10:44.190685 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:44.190797 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:44.190884 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:44.190989 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:44.191120 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:44.191221 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9397d415e86d427d8fe2f9e906b59e2c: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:44.191318 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:44.191409 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:44.191486 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server d3917663631a4479b15355381486591c
W20240501 20:10:44.191588 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:44.191665 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:44.191756 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:44.191843 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:44.254738 18244 rebalancer_tool.cc:1074] tablet b016fc1d28524a02bd762b74c6b08375: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:44.412868 18244 rebalancer_tool.cc:1084] tablet 58ae613944b941e694f790e02cfe5292: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move ignored: Illegal state: tablet 58ae613944b941e694f790e02cfe5292: replica cad00548f33849e7a19cecd760e1e429 is already present
I20240501 20:10:44.412997 18244 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:47.026312 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:47.026455 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:47.026543 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:47.026798 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:47.026875 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
W20240501 20:10:47.027032 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9397d415e86d427d8fe2f9e906b59e2c: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:47.027103 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:47.027173 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
W20240501 20:10:47.027252 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:47.027376 18244 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server cad00548f33849e7a19cecd760e1e429
I20240501 20:10:47.586035 18244 rebalancer_tool.cc:1074] tablet ff11a8e00b544b4193a2824541668492: '042aee20cf4e4e63be3ecc013fda59c9' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:47.861492 18244 rebalancer_tool.cc:1074] tablet c538581c71f646b5bc8d71cb6981f201: '042aee20cf4e4e63be3ecc013fda59c9' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:48.098889 18244 rebalancer_tool.cc:1074] tablet f904b62401ff4528b3fa65dd4e11512a: 'f856e0d1074b431eb19763847958b89e' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:48.361053 18244 rebalancer_tool.cc:1074] tablet 84d7bbca7af34a2db637b55635fa2f7a: '754fcd3b82f041568f546c6205f44272' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:48.585500 18244 rebalancer_tool.cc:1074] tablet f638f341ff2341dc82889334eb87e3d0: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:49.258360 18244 rebalancer_tool.cc:1074] tablet 3643d6995910404fb8a7e6ef9c635f42: '042aee20cf4e4e63be3ecc013fda59c9' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:49.561666 18244 rebalancer_tool.cc:1074] tablet 5682259bc54c469e9c6a08723c1bbee0: 'f856e0d1074b431eb19763847958b89e' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:49.860319 18244 rebalancer_tool.cc:1074] tablet 8f3eb13093b546d38fc71368f2083878: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:49.941895 18244 rebalancer_tool.cc:1084] tablet ddf5ddac08364043bf2e27184d5d3213: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '9a0f0037700246bdb65e05aef95fb2be' move ignored: Illegal state: tablet ddf5ddac08364043bf2e27184d5d3213: replica 9a0f0037700246bdb65e05aef95fb2be is already present
I20240501 20:10:49.942160 18244 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:50.595690 18244 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.640s	user 0.044s	sys 0.095s
W20240501 20:10:50.595921 18244 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.640s	user 0.045s	sys 0.095s
I20240501 20:10:52.946189 18244 rebalancer_tool.cc:1074] tablet ddf5ddac08364043bf2e27184d5d3213: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '754fcd3b82f041568f546c6205f44272' move scheduled
I20240501 20:10:53.194641 18244 rebalancer_tool.cc:1074] tablet f345303c72774848bb60e806cff22d12: 'd3917663631a4479b15355381486591c' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:53.341506 18244 rebalancer_tool.cc:1074] tablet eef58fcca15041a39e685a7cdcb04f2b: '042aee20cf4e4e63be3ecc013fda59c9' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:53.523626 18244 rebalancer_tool.cc:1074] tablet d48de8e0c12b432ea5079390400302c1: 'f856e0d1074b431eb19763847958b89e' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:54.208124 18244 rebalancer_tool.cc:1074] tablet 67d59d1654ef4ab3ac996f55a73b313f: '754fcd3b82f041568f546c6205f44272' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:54.505074 18244 rebalancer_tool.cc:1084] tablet eb90224d08f84a2faf7dcc063c6af56e: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move ignored: Illegal state: tablet eb90224d08f84a2faf7dcc063c6af56e: replica d3917663631a4479b15355381486591c is already present
I20240501 20:10:54.505208 18244 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): unacceptable health status UNAVAILABLE

I20240501 20:10:56.068921 20922 tablet_copy_client.cc:523] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:56.078393 20922 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap starting.
W20240501 20:10:56.089180 15809 consensus_peers.cc:462] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:56.100553 18240 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 20:10:37.644249 18248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:37.644670 18248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:37.656819 18248 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:39.825572 18248 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:10:41.228334 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:41.228555 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:41.228711 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.228928 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.229028 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.229137 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:41.229292 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:41.229429 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:41.229521 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9397d415e86d427d8fe2f9e906b59e2c: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.229635 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:41.229730 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:41.229907 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:41.230044 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
W20240501 20:10:41.230178 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:41.230270 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9397d415e86d427d8fe2f9e906b59e2c: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:41.395387 18248 rebalancer_tool.cc:1074] tablet 2e6b15e4aebf4b1f9a3176f87083703e: 'f856e0d1074b431eb19763847958b89e' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:41.577145 18248 rebalancer_tool.cc:1074] tablet 5682259bc54c469e9c6a08723c1bbee0: '754fcd3b82f041568f546c6205f44272' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:41.706691 18248 rebalancer_tool.cc:1074] tablet 0f25f3a86f0b494b89fe8f86aae4f4cd: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:41.834676 18248 rebalancer_tool.cc:1074] tablet 2fce37f2e8f74a119b8bd0c34b1a35ac: '042aee20cf4e4e63be3ecc013fda59c9' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:42.001720 18248 rebalancer_tool.cc:1074] tablet 627e9e5c120747f1916855ea83008657: 'f856e0d1074b431eb19763847958b89e' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:42.431121 18248 rebalancer_tool.cc:1074] tablet 3643d6995910404fb8a7e6ef9c635f42: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:42.720036 18248 rebalancer_tool.cc:1074] tablet f7fb9bc4cd73492da44bde1eecd627dc: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:42.914667 18248 rebalancer_tool.cc:1074] tablet f904b62401ff4528b3fa65dd4e11512a: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:43.061201 18248 rebalancer_tool.cc:1074] tablet 36465c82164a4765a63590cfd5d413a4: 'f856e0d1074b431eb19763847958b89e' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:43.486028 18248 rebalancer_tool.cc:1074] tablet c538581c71f646b5bc8d71cb6981f201: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:43.659128 18248 rebalancer_tool.cc:1074] tablet f3761f02c20a4448acd89f4a37f1193c: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:43.818497 18248 rebalancer_tool.cc:1074] tablet eb90224d08f84a2faf7dcc063c6af56e: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:44.367707 18248 rebalancer_tool.cc:1074] tablet c74a6da428b04df58b7edcdf88d81cca: 'f856e0d1074b431eb19763847958b89e' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:44.485632 18248 rebalancer_tool.cc:1074] tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b: '042aee20cf4e4e63be3ecc013fda59c9' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:44.812801 18248 rebalancer_tool.cc:1074] tablet d3ba52a871714255aa1bd34d9daca920: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:45.223078 18248 rebalancer_tool.cc:1074] tablet b016fc1d28524a02bd762b74c6b08375: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:45.417495 18248 rebalancer_tool.cc:1074] tablet ddf5ddac08364043bf2e27184d5d3213: 'f856e0d1074b431eb19763847958b89e' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:46.042191 18248 rebalancer_tool.cc:1074] tablet 1474070bbdf147d2ae3313fdf9d71c19: '042aee20cf4e4e63be3ecc013fda59c9' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:46.550069 18248 rebalancer_tool.cc:1074] tablet d48de8e0c12b432ea5079390400302c1: '754fcd3b82f041568f546c6205f44272' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:46.856891 18248 rebalancer_tool.cc:1136] tablet c538581c71f646b5bc8d71cb6981f201: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move completed: OK
I20240501 20:10:47.149485 18248 rebalancer_tool.cc:1136] tablet c74a6da428b04df58b7edcdf88d81cca: 'f856e0d1074b431eb19763847958b89e' -> 'd3917663631a4479b15355381486591c' move completed: OK
I20240501 20:10:47.284708 18248 rebalancer_tool.cc:1136] tablet 627e9e5c120747f1916855ea83008657: 'f856e0d1074b431eb19763847958b89e' -> '9a0f0037700246bdb65e05aef95fb2be' move completed: OK
I20240501 20:10:47.619709 18248 rebalancer_tool.cc:1136] tablet 2fce37f2e8f74a119b8bd0c34b1a35ac: '042aee20cf4e4e63be3ecc013fda59c9' -> 'd3917663631a4479b15355381486591c' move completed: OK
I20240501 20:10:47.899169 18248 rebalancer_tool.cc:1136] tablet 2e6b15e4aebf4b1f9a3176f87083703e: 'f856e0d1074b431eb19763847958b89e' -> 'cad00548f33849e7a19cecd760e1e429' move completed: OK
I20240501 20:10:48.207042 18248 rebalancer_tool.cc:1136] tablet f904b62401ff4528b3fa65dd4e11512a: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move completed: OK
I20240501 20:10:48.654564 18248 rebalancer_tool.cc:1136] tablet 5682259bc54c469e9c6a08723c1bbee0: '754fcd3b82f041568f546c6205f44272' -> '9a0f0037700246bdb65e05aef95fb2be' move completed: OK
I20240501 20:10:49.007867 18248 rebalancer_tool.cc:1136] tablet 0f25f3a86f0b494b89fe8f86aae4f4cd: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '047275bda1074ac5a63325e7e1109c61' move completed: OK
I20240501 20:10:49.252030 18248 rebalancer_tool.cc:1136] tablet f7fb9bc4cd73492da44bde1eecd627dc: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move completed: OK
I20240501 20:10:49.474516 18248 rebalancer_tool.cc:1136] tablet 36465c82164a4765a63590cfd5d413a4: 'f856e0d1074b431eb19763847958b89e' -> '9a0f0037700246bdb65e05aef95fb2be' move completed: OK
W20240501 20:10:50.341009 18248 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.628s	user 0.086s	sys 0.122s
W20240501 20:10:50.341219 18248 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.628s	user 0.086s	sys 0.122s
I20240501 20:10:50.381954 18248 rebalancer_tool.cc:1136] tablet f3761f02c20a4448acd89f4a37f1193c: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move completed: OK
I20240501 20:10:50.568542 18248 rebalancer_tool.cc:1136] tablet 1474070bbdf147d2ae3313fdf9d71c19: '042aee20cf4e4e63be3ecc013fda59c9' -> 'cad00548f33849e7a19cecd760e1e429' move completed: OK
I20240501 20:10:50.749822 18248 rebalancer_tool.cc:1136] tablet eb90224d08f84a2faf7dcc063c6af56e: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move completed: OK
I20240501 20:10:50.894246 18248 rebalancer_tool.cc:1136] tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b: '042aee20cf4e4e63be3ecc013fda59c9' -> '9a0f0037700246bdb65e05aef95fb2be' move completed: Incomplete: tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b, TS 042aee20cf4e4e63be3ecc013fda59c9 -> TS 9a0f0037700246bdb65e05aef95fb2be move failed, target replica disappeared
I20240501 20:10:50.992300 18248 rebalancer_tool.cc:1136] tablet b016fc1d28524a02bd762b74c6b08375: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move completed: OK
I20240501 20:10:51.203330 18248 rebalancer_tool.cc:1136] tablet 3643d6995910404fb8a7e6ef9c635f42: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move completed: OK
I20240501 20:10:51.284426 18248 rebalancer_tool.cc:1136] tablet d3ba52a871714255aa1bd34d9daca920: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move completed: OK
I20240501 20:10:51.509826 18248 rebalancer_tool.cc:1074] tablet 8f3eb13093b546d38fc71368f2083878: '042aee20cf4e4e63be3ecc013fda59c9' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:51.703101 18248 rebalancer_tool.cc:1074] tablet 98eb95ded28d433aad9523586e29947f: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:51.805806 18248 rebalancer_tool.cc:1084] tablet f345303c72774848bb60e806cff22d12: 'f856e0d1074b431eb19763847958b89e' -> 'd3917663631a4479b15355381486591c' move ignored: Illegal state: tablet f345303c72774848bb60e806cff22d12: replica d3917663631a4479b15355381486591c is already present
I20240501 20:10:51.805977 18248 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:53.551606 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 754fcd3b82f041568f546c6205f44272
W20240501 20:10:53.551949 18248 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:53.754384 18248 rebalancer_tool.cc:1074] tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b: '754fcd3b82f041568f546c6205f44272' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:53.800783 18248 rebalancer_tool.cc:1084] tablet eef58fcca15041a39e685a7cdcb04f2b: 'f856e0d1074b431eb19763847958b89e' -> 'cad00548f33849e7a19cecd760e1e429' move ignored: Illegal state: tablet eef58fcca15041a39e685a7cdcb04f2b: replica cad00548f33849e7a19cecd760e1e429 is already present
I20240501 20:10:53.800920 18248 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:54.693007 20925 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:54.653068 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.200:38791 (local address 127.0.0.1:40638)
0501 20:10:54.671110 (+ 18042us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:54.671127 (+    17us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:54.671227 (+   100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:54.691722 (+ 20495us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":17938,"thread_start_us":11563,"threads_started":1}
Illegal state: tablet server 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): unacceptable health status UNAVAILABLE

I20240501 20:10:56.139034 20922 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:56.139511 20922 tablet_bootstrap.cc:492] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: Bootstrap complete.
I20240501 20:10:56.139794 20922 ts_tablet_manager.cc:1367] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: Time spent bootstrapping tablet: real 0.062s	user 0.040s	sys 0.008s
I20240501 20:10:56.140507 20922 raft_consensus.cc:348] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:56.140763 20922 raft_consensus.cc:729] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a0f0037700246bdb65e05aef95fb2be, State: Initialized, Role: LEARNER
I20240501 20:10:56.140995 20922 consensus_queue.cc:260] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: true } }
I20240501 20:10:56.141649 20922 ts_tablet_manager.cc:1398] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:10:56.142500 16285 tablet_copy_service.cc:342] P f856e0d1074b431eb19763847958b89e: Request end of tablet copy session 9a0f0037700246bdb65e05aef95fb2be-c538581c71f646b5bc8d71cb6981f201 received from {username='slave'} at 127.8.68.197:55615
I20240501 20:10:56.142638 16285 tablet_copy_service.cc:433] P f856e0d1074b431eb19763847958b89e: ending tablet copy session 9a0f0037700246bdb65e05aef95fb2be-c538581c71f646b5bc8d71cb6981f201 on tablet c538581c71f646b5bc8d71cb6981f201 with peer 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:56.187436 21019 raft_consensus.cc:1050] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:56.188323 21019 consensus_queue.cc:237] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:56.195940 17793 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:56.196563 20749 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:10:56.201743 15809 consensus_peers.cc:485] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:56.202961 16003 raft_consensus.cc:1261] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:56.203645 21019 consensus_queue.cc:1035] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:56.224754 21019 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.230809 16003 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.235503 17793 raft_consensus.cc:2932] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.243841 15731 catalog_manager.cc:5484] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be reported cstate change: config changed from index 9 to 10, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.259507 17793 raft_consensus.cc:1203] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 20:10:56.279937 18239 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 20:10:37.887198 18243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:37.887642 18243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:37.899106 18243 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:41.106473 18243 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:10:42.875422 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server d3917663631a4479b15355381486591c
W20240501 20:10:42.875612 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.875721 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
W20240501 20:10:42.875839 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.875937 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:42.876093 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:42.876242 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:42.876354 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.876458 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.876598 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.876721 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:42.876875 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:42.877005 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.877133 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.877238 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
I20240501 20:10:43.289552 18243 rebalancer_tool.cc:1074] tablet 2fce37f2e8f74a119b8bd0c34b1a35ac: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:43.678546 18243 rebalancer_tool.cc:1074] tablet b016fc1d28524a02bd762b74c6b08375: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:44.376977 18243 rebalancer_tool.cc:1074] tablet 58ae613944b941e694f790e02cfe5292: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:44.669898 18243 rebalancer_tool.cc:1074] tablet e483217b8761457d84edc49253877719: 'f856e0d1074b431eb19763847958b89e' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:44.943738 18243 rebalancer_tool.cc:1074] tablet 98eb95ded28d433aad9523586e29947f: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:45.172021 18243 rebalancer_tool.cc:1084] tablet 0f25f3a86f0b494b89fe8f86aae4f4cd: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'cad00548f33849e7a19cecd760e1e429' move ignored: Not found: Peer with uuid 51b81a5ae56b4dd99af1b0167c1da1e2 not found in consensus config
I20240501 20:10:45.172194 18243 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:47.195904 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:47.196156 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:47.196326 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:47.196462 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 047275bda1074ac5a63325e7e1109c61 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:47.196548 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9397d415e86d427d8fe2f9e906b59e2c: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:47.196631 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:47.196712 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
W20240501 20:10:47.196805 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:47.196884 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:47.196977 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:47.447333 18243 rebalancer_tool.cc:1074] tablet 8f3eb13093b546d38fc71368f2083878: '754fcd3b82f041568f546c6205f44272' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:47.671321 18243 rebalancer_tool.cc:1074] tablet 36465c82164a4765a63590cfd5d413a4: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:47.908531 18243 rebalancer_tool.cc:1074] tablet f345303c72774848bb60e806cff22d12: 'f856e0d1074b431eb19763847958b89e' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:48.005687 18243 rebalancer_tool.cc:1074] tablet 3643d6995910404fb8a7e6ef9c635f42: '042aee20cf4e4e63be3ecc013fda59c9' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:48.098464 18243 rebalancer_tool.cc:1074] tablet 8e0746bed2cf43b9bc7765c270a8aa62: '754fcd3b82f041568f546c6205f44272' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:48.729660 18243 rebalancer_tool.cc:1084] tablet f638f341ff2341dc82889334eb87e3d0: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240501 20:10:48.729789 18243 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:50.236616 18243 thread.cc:636] RemoteKsckCluster-fetch [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.112s	sys 0.168s
W20240501 20:10:50.236836 18243 thread.cc:602] RemoteKsckCluster-fetch [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.112s	sys 0.168s
W20240501 20:10:51.633855 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:51.634027 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server d3917663631a4479b15355381486591c
W20240501 20:10:51.649302 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:51.649443 18243 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9397d415e86d427d8fe2f9e906b59e2c: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:51.905655 18243 rebalancer_tool.cc:1074] tablet 2e6b15e4aebf4b1f9a3176f87083703e: 'd3917663631a4479b15355381486591c' -> '' move scheduled
I20240501 20:10:52.302886 18243 rebalancer_tool.cc:1074] tablet c74a6da428b04df58b7edcdf88d81cca: '042aee20cf4e4e63be3ecc013fda59c9' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:52.596987 18243 rebalancer_tool.cc:1074] tablet 627e9e5c120747f1916855ea83008657: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:52.790534 18243 rebalancer_tool.cc:1074] tablet eb90224d08f84a2faf7dcc063c6af56e: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:52.885422 18243 rebalancer_tool.cc:1084] tablet eef58fcca15041a39e685a7cdcb04f2b: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move ignored: Illegal state: tablet eef58fcca15041a39e685a7cdcb04f2b: replica 047275bda1074ac5a63325e7e1109c61 is already present
I20240501 20:10:52.885551 18243 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:54.679548 20777 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:54.118876 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.200:38791 (local address 127.0.0.1:40622)
0501 20:10:54.119074 (+   198us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:54.119094 (+    20us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:54.119216 (+   122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:54.678176 (+558960us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
W20240501 20:10:55.829241 21074 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:55.787049 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.198:44869 (local address 127.0.0.1:58142)
0501 20:10:55.828891 (+ 41842us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:55.828988 (+    97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41730,"threads_started":1}
Illegal state: tablet server 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): unacceptable health status UNAVAILABLE

I20240501 20:10:56.284535 15854 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 20:10:56.292686 16199 consensus_peers.cc:462] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection refused (error 111)
I20240501 20:10:56.330191 17773 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:56.330727 18040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:56.331751 15985 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:56.347681 16244 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:56.348366 16116 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:56.427709 21019 raft_consensus.cc:1050] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2: attempting to promote NON_VOTER cad00548f33849e7a19cecd760e1e429 to VOTER
I20240501 20:10:56.428490 21019 consensus_queue.cc:237] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:56.432742 18060 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 LEARNER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:56.432894 17792 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:56.433472 20749 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:56.433956 20749 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:10:56.434545 15809 consensus_peers.cc:485] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:56.444622 16003 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:56.445274 21125 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:56.446542 21019 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:56.447292 18060 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:56.452236 17792 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:56.453024 15733 catalog_manager.cc:5484] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 8 to 9, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:56.454782 16005 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, cad00548f33849e7a19cecd760e1e429 (127.8.68.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:56.464658 17792 raft_consensus.cc:1203] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240501 20:10:56.466477 15873 consensus_queue.cc:237] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:56.468729 18060 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:56.469214 17793 raft_consensus.cc:1261] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer 51b81a5ae56b4dd99af1b0167c1da1e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:10:56.469722 21017 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:56.470111 21125 consensus_queue.cc:1035] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:56.473806 21017 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:56.474634 18060 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
W20240501 20:10:56.477308 15809 consensus_peers.cc:485] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:56.477770 17793 raft_consensus.cc:2932] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } }
I20240501 20:10:56.479791 15719 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 627e9e5c120747f1916855ea83008657 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:56.480751 15731 catalog_manager.cc:5484] T 627e9e5c120747f1916855ea83008657 P 51b81a5ae56b4dd99af1b0167c1da1e2 reported cstate change: config changed from index 9 to 10, VOTER 754fcd3b82f041568f546c6205f44272 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:10:56.485361 15809 consensus_peers.cc:462] T e483217b8761457d84edc49253877719 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:56.486116 15985 tablet_service.cc:1514] Processing DeleteTablet for tablet 627e9e5c120747f1916855ea83008657 with delete_type TABLET_DATA_TOMBSTONED (TS 754fcd3b82f041568f546c6205f44272 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44236
I20240501 20:10:56.489607 21131 tablet_replica.cc:330] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272: stopping tablet replica
I20240501 20:10:56.489835 21131 raft_consensus.cc:2229] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:56.490386 21131 raft_consensus.cc:2258] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:56.491289 21131 ts_tablet_manager.cc:1875] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:56.497967 21131 ts_tablet_manager.cc:1888] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:10:56.498106 21131 log.cc:1196] T 627e9e5c120747f1916855ea83008657 P 754fcd3b82f041568f546c6205f44272: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/627e9e5c120747f1916855ea83008657
I20240501 20:10:56.498863 15720 catalog_manager.cc:4830] TS 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): tablet 627e9e5c120747f1916855ea83008657 (table rebalance_test_table_2 [id=db16b6d911694b0486d5a896b8306375]) successfully deleted
W20240501 20:10:56.505602 15808 consensus_peers.cc:485] T d48de8e0c12b432ea5079390400302c1 P 51b81a5ae56b4dd99af1b0167c1da1e2 -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:56.614270 18241 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 20:10:37.958737 18246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:37.959108 18246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:37.971235 18246 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:41.198925 18246 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:10:42.956099 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server d3917663631a4479b15355381486591c
W20240501 20:10:42.956419 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:42.956530 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.956611 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.956735 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.956813 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:42.956985 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:42.957116 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:42.957278 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.957360 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.957481 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.957569 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:42.957691 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:42.957777 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:42.957873 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server d3917663631a4479b15355381486591c
I20240501 20:10:43.575081 18246 rebalancer_tool.cc:1074] tablet 67d59d1654ef4ab3ac996f55a73b313f: '51b81a5ae56b4dd99af1b0167c1da1e2' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:43.844252 18246 rebalancer_tool.cc:1074] tablet d9a1a818ab7e44a5ac5c67f42e1dfe3b: '042aee20cf4e4e63be3ecc013fda59c9' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:43.960079 18246 rebalancer_tool.cc:1084] tablet c538581c71f646b5bc8d71cb6981f201: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move ignored: Illegal state: tablet c538581c71f646b5bc8d71cb6981f201: replica cad00548f33849e7a19cecd760e1e429 is already present
I20240501 20:10:43.960254 18246 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:46.745568 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:46.745707 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa7a41060cc1492b8f99aac4eedf2e0f: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
W20240501 20:10:46.745790 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
W20240501 20:10:46.746131 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:46.746218 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 623ab7f13c234a3ab8bf37522d87130c: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:46.746464 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table db16b6d911694b0486d5a896b8306375: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server 9a0f0037700246bdb65e05aef95fb2be
W20240501 20:10:46.746582 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6757c3c851a94b17ae18e261f7bb22d5: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server 047275bda1074ac5a63325e7e1109c61
W20240501 20:10:46.749819 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1335198d9bd43dcbdaeb39d406d227b: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server d3917663631a4479b15355381486591c
W20240501 20:10:46.749959 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:46.750100 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c26f53ebc0e84a86904dd9a39a23b960: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:47.038251 18246 rebalancer_tool.cc:1074] tablet 8e0746bed2cf43b9bc7765c270a8aa62: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:47.230415 18246 rebalancer_tool.cc:1074] tablet 1474070bbdf147d2ae3313fdf9d71c19: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:47.343029 18246 rebalancer_tool.cc:1084] tablet 36465c82164a4765a63590cfd5d413a4: 'f856e0d1074b431eb19763847958b89e' -> '047275bda1074ac5a63325e7e1109c61' move ignored: Not found: Peer with uuid f856e0d1074b431eb19763847958b89e not found in consensus config
I20240501 20:10:47.343164 18246 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:49.763358 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server 754fcd3b82f041568f546c6205f44272 to server d3917663631a4479b15355381486591c
W20240501 20:10:49.763504 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bdaa80eaeb6c4114930a1001be0cb61d: could not find any suitable replica to move from server 51b81a5ae56b4dd99af1b0167c1da1e2 to server cad00548f33849e7a19cecd760e1e429
W20240501 20:10:50.394703 18246 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.618s	user 0.101s	sys 0.157s
W20240501 20:10:50.394896 18246 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.618s	user 0.101s	sys 0.157s
I20240501 20:10:50.538976 18246 rebalancer_tool.cc:1084] tablet ddf5ddac08364043bf2e27184d5d3213: 'f856e0d1074b431eb19763847958b89e' -> '754fcd3b82f041568f546c6205f44272' move ignored: Not found: Peer with uuid f856e0d1074b431eb19763847958b89e not found in consensus config
I20240501 20:10:50.539126 18246 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:53.133957 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f219e025e794415a83c70d7e68f6b884: could not find any suitable replica to move from server f856e0d1074b431eb19763847958b89e to server 042aee20cf4e4e63be3ecc013fda59c9
W20240501 20:10:53.134246 18246 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 322459d71dfc45c7b076dfc0b0076901: could not find any suitable replica to move from server 042aee20cf4e4e63be3ecc013fda59c9 to server 047275bda1074ac5a63325e7e1109c61
I20240501 20:10:53.299314 18246 rebalancer_tool.cc:1074] tablet 3643d6995910404fb8a7e6ef9c635f42: 'f856e0d1074b431eb19763847958b89e' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:53.450563 18246 rebalancer_tool.cc:1074] tablet e483217b8761457d84edc49253877719: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '047275bda1074ac5a63325e7e1109c61' move scheduled
I20240501 20:10:53.803627 18246 rebalancer_tool.cc:1074] tablet d3ba52a871714255aa1bd34d9daca920: '042aee20cf4e4e63be3ecc013fda59c9' -> 'd3917663631a4479b15355381486591c' move scheduled
I20240501 20:10:54.156208 18246 rebalancer_tool.cc:1074] tablet 58ae613944b941e694f790e02cfe5292: '042aee20cf4e4e63be3ecc013fda59c9' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:54.977957 18246 rebalancer_tool.cc:1074] tablet 627e9e5c120747f1916855ea83008657: '754fcd3b82f041568f546c6205f44272' -> 'cad00548f33849e7a19cecd760e1e429' move scheduled
I20240501 20:10:55.508101 18246 rebalancer_tool.cc:1074] tablet c538581c71f646b5bc8d71cb6981f201: 'd3917663631a4479b15355381486591c' -> '9a0f0037700246bdb65e05aef95fb2be' move scheduled
I20240501 20:10:55.692709 18246 rebalancer_tool.cc:1084] tablet ddf5ddac08364043bf2e27184d5d3213: '51b81a5ae56b4dd99af1b0167c1da1e2' -> '754fcd3b82f041568f546c6205f44272' move ignored: Illegal state: tablet ddf5ddac08364043bf2e27184d5d3213: replica 754fcd3b82f041568f546c6205f44272 is already present
I20240501 20:10:55.692859 18246 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): unacceptable health status UNAVAILABLE

E20240501 20:10:56.614810  8467 external_mini_cluster.cc:943] Process with UUID d3917663631a4479b15355381486591c has crashed
E20240501 20:10:56.614914  8467 external_mini_cluster.cc:943] Process with UUID 047275bda1074ac5a63325e7e1109c61 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240501 20:10:56.616511  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 15793
I20240501 20:10:56.623219 15915 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:56.708178 16199 consensus_peers.cc:462] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection refused (error 111)
I20240501 20:10:56.836934 20692 raft_consensus.cc:1050] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
I20240501 20:10:56.837751 20692 consensus_queue.cc:237] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:56.840493 18060 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:56.840667 17792 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:56.841041 16263 raft_consensus.cc:1261] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Refusing update from remote peer 042aee20cf4e4e63be3ecc013fda59c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:56.841240 21030 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:56.841768 21030 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:56.842257 21030 consensus_queue.cc:1035] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:56.850184 21008 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.855564 15731 catalog_manager.cc:5484] T 58ae613944b941e694f790e02cfe5292 P 042aee20cf4e4e63be3ecc013fda59c9 reported cstate change: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "042aee20cf4e4e63be3ecc013fda59c9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:56.858045 18058 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.860467 17792 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.864084 16263 raft_consensus.cc:2932] T 58ae613944b941e694f790e02cfe5292 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.888442  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 15793
W20240501 20:10:56.923987 17994 connection.cc:535] server connection from 127.8.68.193:37139 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:56.924448  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 15924
W20240501 20:10:56.924542 16200 connection.cc:535] client connection to 127.8.68.193:35513 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:56.925480 16200 consensus_peers.cc:485] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e -> Peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Couldn't send request to peer 51b81a5ae56b4dd99af1b0167c1da1e2. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:56.925961 16070 consensus_peers.cc:485] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Couldn't send request to peer 51b81a5ae56b4dd99af1b0167c1da1e2. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:56.930639 16043 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:56.933784 20159 raft_consensus.cc:1050] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e: attempting to promote NON_VOTER 9a0f0037700246bdb65e05aef95fb2be to VOTER
W20240501 20:10:56.934192 15719 catalog_manager.cc:4573] TS 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eef58fcca15041a39e685a7cdcb04f2b: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:56.934679 20159 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:56.936807 17792 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 LEARNER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:56.937093 18058 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:10:56.937350 20820 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:10:56.937806 20820 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:10:56.938855 16199 consensus_peers.cc:485] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Couldn't send request to peer d3917663631a4479b15355381486591c. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:56.941112 20820 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.941758 18060 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.945966 17792 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.946807 15734 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 reported cstate change: config changed from index 10 to 11, 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: false replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.953125 16263 consensus_queue.cc:237] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:56.954696 17792 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:10:56.954743 18060 raft_consensus.cc:1261] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Refusing update from remote peer f856e0d1074b431eb19763847958b89e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:10:56.955191 21147 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:10:56.955536 20820 consensus_queue.cc:1035] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:10:56.957892 20159 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.958820 17792 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.958851 18058 raft_consensus.cc:2932] T c538581c71f646b5bc8d71cb6981f201 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } }
I20240501 20:10:56.961207 15718 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c538581c71f646b5bc8d71cb6981f201 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:56.963340 15731 catalog_manager.cc:5484] T c538581c71f646b5bc8d71cb6981f201 P f856e0d1074b431eb19763847958b89e reported cstate change: config changed from index 11 to 12, VOTER d3917663631a4479b15355381486591c (127.8.68.198) evicted. New cstate: current_term: 1 leader_uuid: "f856e0d1074b431eb19763847958b89e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:10:57.145408 16200 consensus_peers.cc:485] T eb90224d08f84a2faf7dcc063c6af56e P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:57.146870 16070 consensus_peers.cc:485] T 84d7bbca7af34a2db637b55635fa2f7a P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Couldn't send request to peer 51b81a5ae56b4dd99af1b0167c1da1e2. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:57.206588 16199 consensus_peers.cc:462] T d3ba52a871714255aa1bd34d9daca920 P f856e0d1074b431eb19763847958b89e -> Peer d3917663631a4479b15355381486591c (127.8.68.198:44869): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.68.198:44869: connect: Connection refused (error 111)
I20240501 20:10:57.207547  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 15924
W20240501 20:10:57.242156 16071 connection.cc:535] client connection to 127.8.68.194:46677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:57.242602  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 16054
W20240501 20:10:57.243227 16071 consensus_peers.cc:485] T ff11a8e00b544b4193a2824541668492 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Couldn't send request to peer 754fcd3b82f041568f546c6205f44272. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:46677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:57.243356 16071 consensus_peers.cc:485] T eef58fcca15041a39e685a7cdcb04f2b P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Couldn't send request to peer 754fcd3b82f041568f546c6205f44272. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:46677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:57.243490 16071 consensus_peers.cc:485] T f345303c72774848bb60e806cff22d12 P 042aee20cf4e4e63be3ecc013fda59c9 -> Peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Couldn't send request to peer 754fcd3b82f041568f546c6205f44272. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:46677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:57.248692 16174 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:57.355835 16200 consensus_peers.cc:485] T 98eb95ded28d433aad9523586e29947f P f856e0d1074b431eb19763847958b89e -> Peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Couldn't send request to peer 047275bda1074ac5a63325e7e1109c61. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:57.481343  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 16054
W20240501 20:10:57.514830 17993 connection.cc:535] server connection from 127.8.68.195:55659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:57.514835 17730 connection.cc:535] server connection from 127.8.68.195:36333 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:57.515321  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 16184
I20240501 20:10:57.521090 16305 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:57.736194 21157 raft_consensus.cc:482] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:57.736447 21157 raft_consensus.cc:504] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "754fcd3b82f041568f546c6205f44272" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 46677 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } }
I20240501 20:10:57.738143 21157 leader_election.cc:288] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677), 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791)
W20240501 20:10:57.739710 17728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
W20240501 20:10:57.741528 17729 leader_election.cc:334] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 754fcd3b82f041568f546c6205f44272 (127.8.68.194:46677): Network error: Client connection negotiation failed: client connection to 127.8.68.194:46677: connect: Connection refused (error 111)
W20240501 20:10:57.741724 17728 leader_election.cc:334] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
I20240501 20:10:57.741869 17728 leader_election.cc:302] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9a0f0037700246bdb65e05aef95fb2be; no voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 754fcd3b82f041568f546c6205f44272
I20240501 20:10:57.742162 21157 raft_consensus.cc:2726] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:57.742607 17728 leader_election.cc:334] T d9a1a818ab7e44a5ac5c67f42e1dfe3b P 9a0f0037700246bdb65e05aef95fb2be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:57.756251 20820 raft_consensus.cc:482] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:57.756446 20820 raft_consensus.cc:504] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "f856e0d1074b431eb19763847958b89e" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 36919 } attrs { replace: true } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } } peers { permanent_uuid: "d3917663631a4479b15355381486591c" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 44869 } attrs { promote: true } }
I20240501 20:10:57.757164 20820 leader_election.cc:288] T d48de8e0c12b432ea5079390400302c1 P f856e0d1074b431eb19763847958b89e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791)
I20240501 20:10:57.757464  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 16184
W20240501 20:10:57.790810 17993 connection.cc:535] server connection from 127.8.68.196:43175 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:57.790858 17728 connection.cc:535] server connection from 127.8.68.196:56775 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:57.791457  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 17708
I20240501 20:10:57.838510 17834 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:57.969867 15719 catalog_manager.cc:4573] TS 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eef58fcca15041a39e685a7cdcb04f2b: Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:57.975878 21165 raft_consensus.cc:482] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:57.976132 21165 raft_consensus.cc:504] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:57.977870 21165 leader_election.cc:288] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623)
W20240501 20:10:57.978811 17993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
W20240501 20:10:57.980043 17993 leader_election.cc:334] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
I20240501 20:10:58.025184 21157 raft_consensus.cc:482] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:58.025382 21157 raft_consensus.cc:504] T 8f3eb13093b546d38fc71368f2083878 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:58.025774 21170 raft_consensus.cc:482] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:58.026247  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 17708
I20240501 20:10:58.025964 21170 raft_consensus.cc:504] T 627e9e5c120747f1916855ea83008657 P 9a0f0037700246bdb65e05aef95fb2be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: NON_VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: true } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:58.051324 21165 raft_consensus.cc:482] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:58.051573 21165 raft_consensus.cc:504] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } attrs { replace: true } } peers { permanent_uuid: "042aee20cf4e4e63be3ecc013fda59c9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 46553 } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } } peers { permanent_uuid: "047275bda1074ac5a63325e7e1109c61" member_type: VOTER last_known_addr { host: "127.8.68.200" port: 38791 } attrs { promote: false } }
I20240501 20:10:58.052675 21165 leader_election.cc:288] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513), 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553), 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791)
W20240501 20:10:58.054333 17994 leader_election.cc:334] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 042aee20cf4e4e63be3ecc013fda59c9 (127.8.68.195:46553): Network error: Client connection negotiation failed: client connection to 127.8.68.195:46553: connect: Connection refused (error 111)
W20240501 20:10:58.054667 17993 leader_election.cc:334] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51b81a5ae56b4dd99af1b0167c1da1e2 (127.8.68.193:35513): Network error: Client connection negotiation failed: client connection to 127.8.68.193:35513: connect: Connection refused (error 111)
I20240501 20:10:58.054806 17993 leader_election.cc:302] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cad00548f33849e7a19cecd760e1e429; no voters: 042aee20cf4e4e63be3ecc013fda59c9, 51b81a5ae56b4dd99af1b0167c1da1e2
W20240501 20:10:58.055150 17993 leader_election.cc:334] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 047275bda1074ac5a63325e7e1109c61 (127.8.68.200:38791): Network error: Client connection negotiation failed: client connection to 127.8.68.200:38791: connect: Connection refused (error 111)
I20240501 20:10:58.055189 21165 raft_consensus.cc:2726] T 2e6b15e4aebf4b1f9a3176f87083703e P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:58.055977  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 17977
W20240501 20:10:58.056521 21168 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:57.978071 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.68.197:37623 (local address 127.8.68.199:45473)
0501 20:10:57.978656 (+   585us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:57.978673 (+    17us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:57.978781 (+   108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:58.026431 (+ 47650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:10:58.026437 (+     6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:10:58.026466 (+    29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:10:58.027126 (+   660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:10:58.027142 (+    16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:10:58.055769 (+ 28627us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.68.197:37623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":406,"threads_started":1}
W20240501 20:10:58.057752 17995 leader_election.cc:334] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a0f0037700246bdb65e05aef95fb2be (127.8.68.197:37623): Network error: Client connection negotiation failed: client connection to 127.8.68.197:37623: connect: Connection refused (error 111)
I20240501 20:10:58.057865 17995 leader_election.cc:302] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cad00548f33849e7a19cecd760e1e429; no voters: 51b81a5ae56b4dd99af1b0167c1da1e2, 9a0f0037700246bdb65e05aef95fb2be
I20240501 20:10:58.058180 21165 raft_consensus.cc:2726] T 627e9e5c120747f1916855ea83008657 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:58.061820 18099 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:58.255579 21165 raft_consensus.cc:482] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51b81a5ae56b4dd99af1b0167c1da1e2)
I20240501 20:10:58.255807 21165 raft_consensus.cc:504] T 8f3eb13093b546d38fc71368f2083878 P cad00548f33849e7a19cecd760e1e429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "51b81a5ae56b4dd99af1b0167c1da1e2" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 35513 } } peers { permanent_uuid: "9a0f0037700246bdb65e05aef95fb2be" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 37623 } attrs { promote: false } } peers { permanent_uuid: "cad00548f33849e7a19cecd760e1e429" member_type: VOTER last_known_addr { host: "127.8.68.199" port: 44275 } attrs { promote: false } }
I20240501 20:10:58.256140  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 17977
I20240501 20:10:58.283211  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 15703
I20240501 20:10:58.284224 15764 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:58.525219  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 15703
2024-05-01T20:10:58Z chronyd exiting
I20240501 20:10:58.576020  8467 test_util.cc:175] -----------------------------------------------
I20240501 20:10:58.576102  8467 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714594130440066-8467-0
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00> (44870 ms)
[----------] 1 test from ConcurrentRebalancersTest (44870 ms total)

[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN      ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20240501 20:10:58.578058  8467 test_util.cc:268] Using random seed: 364341878
I20240501 20:10:58.579769  8467 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:10:58.579823  8467 ts_itest-base.cc:113] --------------
I20240501 20:10:58.579859  8467 ts_itest-base.cc:114] 4 tablet servers
I20240501 20:10:58.579910  8467 ts_itest-base.cc:115] 1 replicas per TS
I20240501 20:10:58.579957  8467 ts_itest-base.cc:116] --------------
2024-05-01T20:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:10:58Z Disabled control of system clock
I20240501 20:10:58.623513  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:43769
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:43769
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240501 20:10:58.935207 21183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:58.935451 21183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:58.935534 21183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:58.942507 21183 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:58.942601 21183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:10:58.942651 21183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:58.942705 21183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:10:58.942749 21183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:10:58.951189 21183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:43769
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:43769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:58.952221 21183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:58.953709 21183 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:58.963706 21188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:58.964119 21183 server_base.cc:1034] running on GCE node
W20240501 20:10:58.964594 21189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:58.964993 21191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:58.965680 21183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:58.966930 21183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:58.968120 21183 hybrid_clock.cc:648] HybridClock initialized: now 1714594258968073 us; error 38 us; skew 500 ppm
I20240501 20:10:58.968423 21183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:58.974742 21183 webserver.cc:460] Webserver started at http://127.8.68.254:39115/ using document root <none> and password file <none>
I20240501 20:10:58.975416 21183 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:58.975518 21183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:58.975827 21183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:58.978221 21183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "306de9831dd043089fcfd8f0053ea7f6"
format_stamp: "Formatted at 2024-05-01 20:10:58 on dist-test-slave-08p9"
I20240501 20:10:58.978991 21183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "306de9831dd043089fcfd8f0053ea7f6"
format_stamp: "Formatted at 2024-05-01 20:10:58 on dist-test-slave-08p9"
I20240501 20:10:58.983659 21183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:10:58.986696 21197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:58.987843 21183 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20240501 20:10:58.987988 21183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "306de9831dd043089fcfd8f0053ea7f6"
format_stamp: "Formatted at 2024-05-01 20:10:58 on dist-test-slave-08p9"
I20240501 20:10:58.988193 21183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:59.005435 21183 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:59.006119 21183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:59.006384 21183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:59.022567 21183 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:43769
I20240501 20:10:59.022595 21248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:43769 every 8 connection(s)
I20240501 20:10:59.023730 21183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:10:59.025017  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 21183
I20240501 20:10:59.025254  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:10:59.027770 21249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.038728 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6: Bootstrap starting.
I20240501 20:10:59.041263 21249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.042130 21249 log.cc:824] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:59.044204 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6: No bootstrap required, opened a new log
I20240501 20:10:59.049707 21249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306de9831dd043089fcfd8f0053ea7f6" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43769 } }
I20240501 20:10:59.050036 21249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.050140 21249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306de9831dd043089fcfd8f0053ea7f6, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.050666 21249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306de9831dd043089fcfd8f0053ea7f6" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43769 } }
I20240501 20:10:59.050840 21249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:10:59.050959 21249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:10:59.051092 21249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.053066 21249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306de9831dd043089fcfd8f0053ea7f6" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43769 } }
I20240501 20:10:59.053467 21249 leader_election.cc:302] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 306de9831dd043089fcfd8f0053ea7f6; no voters: 
I20240501 20:10:59.053959 21249 leader_election.cc:288] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:10:59.054100 21254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:59.054708 21254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [term 1 LEADER]: Becoming Leader. State: Replica: 306de9831dd043089fcfd8f0053ea7f6, State: Running, Role: LEADER
I20240501 20:10:59.055178 21254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306de9831dd043089fcfd8f0053ea7f6" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43769 } }
I20240501 20:10:59.055606 21249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:10:59.057468 21256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 306de9831dd043089fcfd8f0053ea7f6. Latest consensus state: current_term: 1 leader_uuid: "306de9831dd043089fcfd8f0053ea7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306de9831dd043089fcfd8f0053ea7f6" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43769 } } }
I20240501 20:10:59.057802 21256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [sys.catalog]: This master's current role is: LEADER
I20240501 20:10:59.058169 21255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "306de9831dd043089fcfd8f0053ea7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306de9831dd043089fcfd8f0053ea7f6" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 43769 } } }
I20240501 20:10:59.058454 21255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6 [sys.catalog]: This master's current role is: LEADER
I20240501 20:10:59.058734 21259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:10:59.063973 21259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:10:59.072116 21259 catalog_manager.cc:1347] Generated new cluster ID: 0a1018180cf94705981fc78cbcc63a05
I20240501 20:10:59.072219 21259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:10:59.085417 21259 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:10:59.086961 21259 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:10:59.100827 21259 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 306de9831dd043089fcfd8f0053ea7f6: Generated new TSK 0
I20240501 20:10:59.101646 21259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:10:59.116411  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43769
--builtin_ntp_servers=127.8.68.212:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:10:59.446142 21273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:59.446424 21273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:59.446534 21273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:59.453692 21273 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:59.453786 21273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:59.453894 21273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240501 20:10:59.462463 21273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:59.463639 21273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:59.465198 21273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:59.476207 21281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:59.476590 21279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:59.477216 21278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:59.477716 21273 server_base.cc:1034] running on GCE node
I20240501 20:10:59.478236 21273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:59.479148 21273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:59.480363 21273 hybrid_clock.cc:648] HybridClock initialized: now 1714594259480314 us; error 38 us; skew 500 ppm
I20240501 20:10:59.480639 21273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:59.482972 21273 webserver.cc:460] Webserver started at http://127.8.68.193:33757/ using document root <none> and password file <none>
I20240501 20:10:59.483719 21273 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:59.483850 21273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:59.484184 21273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:59.486653 21273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f6d2ed84d9904640bd42888b0c5779c9"
format_stamp: "Formatted at 2024-05-01 20:10:59 on dist-test-slave-08p9"
I20240501 20:10:59.487407 21273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f6d2ed84d9904640bd42888b0c5779c9"
format_stamp: "Formatted at 2024-05-01 20:10:59 on dist-test-slave-08p9"
I20240501 20:10:59.492317 21273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240501 20:10:59.495362 21287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:59.496469 21273 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:10:59.496615 21273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f6d2ed84d9904640bd42888b0c5779c9"
format_stamp: "Formatted at 2024-05-01 20:10:59 on dist-test-slave-08p9"
I20240501 20:10:59.496788 21273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:59.522485 21273 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:59.523186 21273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:59.523452 21273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:59.524309 21273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:59.525621 21273 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:59.525714 21273 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:59.525818 21273 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:59.525888 21273 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:59.555235 21273 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:42333
I20240501 20:10:59.555325 21400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:42333 every 8 connection(s)
I20240501 20:10:59.556594 21273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:10:59.564654  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 21273
I20240501 20:10:59.564857  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:10:59.568542  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43769
--builtin_ntp_servers=127.8.68.212:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:59.569010 21401 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43769
I20240501 20:10:59.569267 21401 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:59.570058 21401 heartbeater.cc:511] Master 127.8.68.254:43769 requested a full tablet report, sending...
I20240501 20:10:59.571924 21214 ts_manager.cc:194] Registered new tserver with Master: f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333)
I20240501 20:10:59.573379 21214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:38067
W20240501 20:10:59.881669 21406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:59.881891 21406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:59.881989 21406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:59.889034 21406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:59.889132 21406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:59.889240 21406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240501 20:10:59.897572 21406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:59.898595 21406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:59.900061 21406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:59.910924 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:59.911496 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:59.911857 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:59.912380 21406 server_base.cc:1034] running on GCE node
I20240501 20:10:59.912868 21406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:59.913647 21406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:59.914834 21406 hybrid_clock.cc:648] HybridClock initialized: now 1714594259914796 us; error 31 us; skew 500 ppm
I20240501 20:10:59.915138 21406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:59.917011 21406 webserver.cc:460] Webserver started at http://127.8.68.194:46173/ using document root <none> and password file <none>
I20240501 20:10:59.917631 21406 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:59.917730 21406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:59.918012 21406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:59.920290 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
format_stamp: "Formatted at 2024-05-01 20:10:59 on dist-test-slave-08p9"
I20240501 20:10:59.920965 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
format_stamp: "Formatted at 2024-05-01 20:10:59 on dist-test-slave-08p9"
I20240501 20:10:59.925418 21406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20240501 20:10:59.929244 21421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:59.930807 21406 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:10:59.930963 21406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
format_stamp: "Formatted at 2024-05-01 20:10:59 on dist-test-slave-08p9"
I20240501 20:10:59.931170 21406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:59.956025 21406 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:10:59.956676 21406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:59.956918 21406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:59.957746 21406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:59.959048 21406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:59.959131 21406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:59.959225 21406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:59.959290 21406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:10:59.987748 21406 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:38847
I20240501 20:10:59.987812 21535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:38847 every 8 connection(s)
I20240501 20:10:59.988814 21406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:10:59.996189  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 21406
I20240501 20:10:59.996354  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:10:59.999487  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43769
--builtin_ntp_servers=127.8.68.212:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:00.000610 21536 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43769
I20240501 20:11:00.000846 21536 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:00.001556 21536 heartbeater.cc:511] Master 127.8.68.254:43769 requested a full tablet report, sending...
I20240501 20:11:00.002779 21214 ts_manager.cc:194] Registered new tserver with Master: 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847)
I20240501 20:11:00.003468 21214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:58291
W20240501 20:11:00.317186 21539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:00.317456 21539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:00.317562 21539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:00.324554 21539 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:00.324656 21539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:00.324769 21539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240501 20:11:00.333055 21539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:00.334161 21539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:00.335783 21539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:00.346967 21544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:00.347433 21545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:00.347827 21539 server_base.cc:1034] running on GCE node
W20240501 20:11:00.348356 21547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:00.348906 21539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:00.349740 21539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:00.350924 21539 hybrid_clock.cc:648] HybridClock initialized: now 1714594260350883 us; error 35 us; skew 500 ppm
I20240501 20:11:00.351204 21539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:00.353220 21539 webserver.cc:460] Webserver started at http://127.8.68.195:38371/ using document root <none> and password file <none>
I20240501 20:11:00.353863 21539 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:00.353965 21539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:00.354244 21539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:00.356683 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "12258cbda618428d9089acc34e1565d3"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-08p9"
I20240501 20:11:00.357390 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "12258cbda618428d9089acc34e1565d3"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-08p9"
I20240501 20:11:00.362488 21539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20240501 20:11:00.365552 21553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:00.366847 21539 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:11:00.366986 21539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "12258cbda618428d9089acc34e1565d3"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-08p9"
I20240501 20:11:00.367170 21539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:00.414090 21539 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:00.414840 21539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:00.415066 21539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:00.415885 21539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:00.417212 21539 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:00.417296 21539 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:00.417373 21539 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:00.417425 21539 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:00.445163 21539 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:34785
I20240501 20:11:00.445221 21666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:34785 every 8 connection(s)
I20240501 20:11:00.446415 21539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:11:00.446828  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 21539
I20240501 20:11:00.447001  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:11:00.450788  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43769
--builtin_ntp_servers=127.8.68.212:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:00.458550 21667 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43769
I20240501 20:11:00.458794 21667 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:00.459489 21667 heartbeater.cc:511] Master 127.8.68.254:43769 requested a full tablet report, sending...
I20240501 20:11:00.460834 21214 ts_manager.cc:194] Registered new tserver with Master: 12258cbda618428d9089acc34e1565d3 (127.8.68.195:34785)
I20240501 20:11:00.461807 21214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:56635
I20240501 20:11:00.575933 21401 heartbeater.cc:503] Master 127.8.68.254:43769 was elected leader, sending a full tablet report...
W20240501 20:11:00.769102 21670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:00.769352 21670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:00.769483 21670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:00.777679 21670 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:00.777807 21670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:00.777982 21670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:11:00.790977 21670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:00.792608 21670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:00.794858 21670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:00.807725 21675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:00.814955 21676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:00.815361 21678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:00.815912 21670 server_base.cc:1034] running on GCE node
I20240501 20:11:00.816527 21670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:00.817553 21670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:00.818756 21670 hybrid_clock.cc:648] HybridClock initialized: now 1714594260818684 us; error 64 us; skew 500 ppm
I20240501 20:11:00.819054 21670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:00.821591 21670 webserver.cc:460] Webserver started at http://127.8.68.196:44839/ using document root <none> and password file <none>
I20240501 20:11:00.822438 21670 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:00.822536 21670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:00.822844 21670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:00.825279 21670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f5235e7de11c49d8b85ebeb503eeaeae"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-08p9"
I20240501 20:11:00.825991 21670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f5235e7de11c49d8b85ebeb503eeaeae"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-08p9"
I20240501 20:11:00.830632 21670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20240501 20:11:00.833612 21684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:00.834789 21670 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:00.834918 21670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f5235e7de11c49d8b85ebeb503eeaeae"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-08p9"
I20240501 20:11:00.835083 21670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:00.864333 21670 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:00.865021 21670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:00.865264 21670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:00.866078 21670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:00.867458 21670 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:00.867542 21670 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:00.867645 21670 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:00.867707 21670 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:00.897050 21670 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:44331
I20240501 20:11:00.897135 21803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:44331 every 8 connection(s)
I20240501 20:11:00.898182 21670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:00.900844  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 21670
I20240501 20:11:00.901012  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:11:00.909644 21804 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43769
I20240501 20:11:00.909893 21804 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:00.910621 21804 heartbeater.cc:511] Master 127.8.68.254:43769 requested a full tablet report, sending...
I20240501 20:11:00.911782 21214 ts_manager.cc:194] Registered new tserver with Master: f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196:44331)
I20240501 20:11:00.912508 21214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:50869
I20240501 20:11:00.916162  8467 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:11:00.929572 21214 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58054:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240501 20:11:00.954057 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 837412d5b37c4b1791f56e6248283aad (DEFAULT_TABLE table=TestTable [id=7115296f422a48d2b68808bee3e3d409]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.955224 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837412d5b37c4b1791f56e6248283aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.964736 21818 tablet_bootstrap.cc:492] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: Bootstrap starting.
I20240501 20:11:00.967602 21818 tablet_bootstrap.cc:654] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.968492 21818 log.cc:824] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:00.970548 21818 tablet_bootstrap.cc:492] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: No bootstrap required, opened a new log
I20240501 20:11:00.970758 21818 ts_tablet_manager.cc:1367] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: Time spent bootstrapping tablet: real 0.006s	user 0.006s	sys 0.000s
I20240501 20:11:00.976493 21818 raft_consensus.cc:348] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:00.976766 21818 raft_consensus.cc:374] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.976840 21818 raft_consensus.cc:729] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.977324 21818 consensus_queue.cc:260] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:00.977496 21818 raft_consensus.cc:388] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:00.977609 21818 raft_consensus.cc:482] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:00.977727 21818 raft_consensus.cc:3037] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.979549 21818 raft_consensus.cc:504] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:00.979947 21818 leader_election.cc:302] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:00.980437 21818 leader_election.cc:288] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:00.980613 21820 raft_consensus.cc:2781] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:00.981228 21820 raft_consensus.cc:686] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:00.981818 21820 consensus_queue.cc:237] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:00.982007 21818 ts_tablet_manager.cc:1398] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.004s
I20240501 20:11:00.986980 21214 catalog_manager.cc:5484] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:00.997345  8467 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:11:00.999053  8467 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f6d2ed84d9904640bd42888b0c5779c9 to finish bootstrapping
I20240501 20:11:01.005460 21536 heartbeater.cc:503] Master 127.8.68.254:43769 was elected leader, sending a full tablet report...
I20240501 20:11:01.006240  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 83b80868c0ff4edabaac58ee1c5b41ac to finish bootstrapping
I20240501 20:11:01.013808  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 12258cbda618428d9089acc34e1565d3 to finish bootstrapping
I20240501 20:11:01.021159  8467 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f5235e7de11c49d8b85ebeb503eeaeae to finish bootstrapping
I20240501 20:11:01.028793  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 21539
I20240501 20:11:01.029582 21662 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:01.106030  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 21539
I20240501 20:11:01.119393  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 21670
I20240501 20:11:01.120441 21799 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:01.199177  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 21670
I20240501 20:11:04.965248 21214 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58054:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:11:04.976306 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 2dff2bedabbf4ae494f535f2e74277cf (DEFAULT_TABLE table=rebalance_test_table_0 [id=529a9c62044e41128f995a109fe9882c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:04.976697 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dff2bedabbf4ae494f535f2e74277cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.982385 21837 tablet_bootstrap.cc:492] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9: Bootstrap starting.
I20240501 20:11:04.984015 21837 tablet_bootstrap.cc:654] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:04.984472 21470 tablet_service.cc:1467] Processing CreateTablet for tablet fad1c1fd383d45dba455731bebfcbbf5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=529a9c62044e41128f995a109fe9882c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:04.984447 21471 tablet_service.cc:1467] Processing CreateTablet for tablet 2fcf8e6204d74bfe9604329491dd5ec3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=529a9c62044e41128f995a109fe9882c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:04.985852 21471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fcf8e6204d74bfe9604329491dd5ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.986129 21837 tablet_bootstrap.cc:492] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9: No bootstrap required, opened a new log
I20240501 20:11:04.986299 21470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad1c1fd383d45dba455731bebfcbbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:04.986271 21837 ts_tablet_manager.cc:1367] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:11:04.987066 21837 raft_consensus.cc:348] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:04.987219 21837 raft_consensus.cc:374] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:04.987318 21837 raft_consensus.cc:729] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Initialized, Role: FOLLOWER
I20240501 20:11:04.987617 21837 consensus_queue.cc:260] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:04.987824 21837 raft_consensus.cc:388] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:04.987906 21837 raft_consensus.cc:482] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:04.987993 21837 raft_consensus.cc:3037] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:04.989921 21837 raft_consensus.cc:504] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:04.990123 21837 leader_election.cc:302] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:04.996673 21839 raft_consensus.cc:2781] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:04.996971 21837 leader_election.cc:288] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:04.997583 21839 raft_consensus.cc:686] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:04.997907 21839 consensus_queue.cc:237] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:04.998934 21837 ts_tablet_manager.cc:1398] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.004s
I20240501 20:11:04.999111 21842 tablet_bootstrap.cc:492] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: Bootstrap starting.
I20240501 20:11:05.000756 21214 catalog_manager.cc:5484] T 2dff2bedabbf4ae494f535f2e74277cf P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:05.002838 21842 tablet_bootstrap.cc:654] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:05.003878 21842 log.cc:824] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:05.006603 21842 tablet_bootstrap.cc:492] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: No bootstrap required, opened a new log
I20240501 20:11:05.006860 21842 ts_tablet_manager.cc:1367] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240501 20:11:05.013350 21842 raft_consensus.cc:348] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:05.013650 21842 raft_consensus.cc:374] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:05.013732 21842 raft_consensus.cc:729] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Initialized, Role: FOLLOWER
I20240501 20:11:05.014245 21842 consensus_queue.cc:260] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:05.014498 21842 raft_consensus.cc:388] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:05.014611 21842 raft_consensus.cc:482] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:05.014739 21842 raft_consensus.cc:3037] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:05.016616 21842 raft_consensus.cc:504] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:05.017035 21842 leader_election.cc:302] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b80868c0ff4edabaac58ee1c5b41ac; no voters: 
I20240501 20:11:05.017558 21842 leader_election.cc:288] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:05.017678 21846 raft_consensus.cc:2781] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:05.018996 21846 raft_consensus.cc:686] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Becoming Leader. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Running, Role: LEADER
I20240501 20:11:05.019241 21842 ts_tablet_manager.cc:1398] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
I20240501 20:11:05.019480 21846 consensus_queue.cc:237] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:05.019734 21842 tablet_bootstrap.cc:492] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac: Bootstrap starting.
I20240501 20:11:05.021174 21842 tablet_bootstrap.cc:654] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:05.023051 21214 catalog_manager.cc:5484] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194). New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:05.026824 21842 tablet_bootstrap.cc:492] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac: No bootstrap required, opened a new log
I20240501 20:11:05.026970 21842 ts_tablet_manager.cc:1367] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.003s
I20240501 20:11:05.027899 21842 raft_consensus.cc:348] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:05.028018 21842 raft_consensus.cc:374] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:05.028096 21842 raft_consensus.cc:729] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Initialized, Role: FOLLOWER
I20240501 20:11:05.028303 21842 consensus_queue.cc:260] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:05.028445 21842 raft_consensus.cc:388] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:05.028515 21842 raft_consensus.cc:482] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:05.028633 21842 raft_consensus.cc:3037] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:05.030534 21842 raft_consensus.cc:504] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:05.030726 21842 leader_election.cc:302] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b80868c0ff4edabaac58ee1c5b41ac; no voters: 
I20240501 20:11:05.030880 21842 leader_election.cc:288] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:05.030946 21846 raft_consensus.cc:2781] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:05.031131 21846 raft_consensus.cc:686] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Becoming Leader. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Running, Role: LEADER
I20240501 20:11:05.031317 21842 ts_tablet_manager.cc:1398] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240501 20:11:05.031347 21846 consensus_queue.cc:237] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:05.033138 21214 catalog_manager.cc:5484] T 2fcf8e6204d74bfe9604329491dd5ec3 P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194). New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.346902 21214 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58054:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:11:06.355199 21336 tablet_service.cc:1467] Processing CreateTablet for tablet c62a19351f9d49ee95909ad405147c04 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f78e9bb52fb643bb8b49d998c92eeb34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:06.355691 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c62a19351f9d49ee95909ad405147c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:06.355576 21471 tablet_service.cc:1467] Processing CreateTablet for tablet e5bd1e1b78f0405d9955bbb4c2ac2e44 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f78e9bb52fb643bb8b49d998c92eeb34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:06.356034 21471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bd1e1b78f0405d9955bbb4c2ac2e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:06.356702 21335 tablet_service.cc:1467] Processing CreateTablet for tablet f6cf57ad231d4706a4a0b62845d90b89 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f78e9bb52fb643bb8b49d998c92eeb34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:06.357081 21335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cf57ad231d4706a4a0b62845d90b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:06.361393 21878 tablet_bootstrap.cc:492] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac: Bootstrap starting.
I20240501 20:11:06.362867 21878 tablet_bootstrap.cc:654] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:06.364133 21879 tablet_bootstrap.cc:492] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9: Bootstrap starting.
I20240501 20:11:06.364586 21878 tablet_bootstrap.cc:492] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac: No bootstrap required, opened a new log
I20240501 20:11:06.364702 21878 ts_tablet_manager.cc:1367] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:11:06.365428 21879 tablet_bootstrap.cc:654] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:06.365370 21878 raft_consensus.cc:348] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.365536 21878 raft_consensus.cc:374] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:06.365643 21878 raft_consensus.cc:729] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Initialized, Role: FOLLOWER
I20240501 20:11:06.365864 21878 consensus_queue.cc:260] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.366001 21878 raft_consensus.cc:388] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:06.366048 21878 raft_consensus.cc:482] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:06.366113 21878 raft_consensus.cc:3037] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:06.366994 21879 tablet_bootstrap.cc:492] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9: No bootstrap required, opened a new log
I20240501 20:11:06.367117 21879 ts_tablet_manager.cc:1367] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:11:06.367736 21879 raft_consensus.cc:348] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.367866 21879 raft_consensus.cc:374] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:06.367949 21879 raft_consensus.cc:729] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Initialized, Role: FOLLOWER
I20240501 20:11:06.368023 21878 raft_consensus.cc:504] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.368238 21878 leader_election.cc:302] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b80868c0ff4edabaac58ee1c5b41ac; no voters: 
I20240501 20:11:06.368300 21879 consensus_queue.cc:260] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.368511 21879 raft_consensus.cc:388] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:06.368603 21879 raft_consensus.cc:482] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:06.368643 21878 leader_election.cc:288] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:06.368719 21879 raft_consensus.cc:3037] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:06.368782 21882 raft_consensus.cc:2781] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:06.370143 21882 raft_consensus.cc:686] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Becoming Leader. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Running, Role: LEADER
I20240501 20:11:06.370486 21882 consensus_queue.cc:237] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.371145 21879 raft_consensus.cc:504] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.371426 21879 leader_election.cc:302] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:06.373006 21214 catalog_manager.cc:5484] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194). New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.379554 21879 leader_election.cc:288] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:06.379717 21886 raft_consensus.cc:2781] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:06.380683 21879 ts_tablet_manager.cc:1398] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9: Time spent starting tablet: real 0.013s	user 0.012s	sys 0.000s
I20240501 20:11:06.381181 21879 tablet_bootstrap.cc:492] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: Bootstrap starting.
I20240501 20:11:06.382417 21879 tablet_bootstrap.cc:654] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:06.382793 21878 ts_tablet_manager.cc:1398] T e5bd1e1b78f0405d9955bbb4c2ac2e44 P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.011s
I20240501 20:11:06.384361 21879 tablet_bootstrap.cc:492] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: No bootstrap required, opened a new log
I20240501 20:11:06.384521 21879 ts_tablet_manager.cc:1367] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:11:06.385149 21879 raft_consensus.cc:348] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.385299 21879 raft_consensus.cc:374] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:06.385378 21879 raft_consensus.cc:729] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Initialized, Role: FOLLOWER
I20240501 20:11:06.385591 21879 consensus_queue.cc:260] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.385727 21879 raft_consensus.cc:388] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:06.385866 21879 raft_consensus.cc:482] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:06.385998 21879 raft_consensus.cc:3037] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:06.387027 21886 raft_consensus.cc:686] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:06.387370 21886 consensus_queue.cc:237] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.388540 21879 raft_consensus.cc:504] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.388756 21879 leader_election.cc:302] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:06.388936 21879 leader_election.cc:288] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:06.389195 21888 raft_consensus.cc:2781] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:06.389349 21879 ts_tablet_manager.cc:1398] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.003s
I20240501 20:11:06.389535 21214 catalog_manager.cc:5484] T c62a19351f9d49ee95909ad405147c04 P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.390149 21888 raft_consensus.cc:686] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:06.390435 21888 consensus_queue.cc:237] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.393460 21214 catalog_manager.cc:5484] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.851279 21214 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58054:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:11:06.858649 21471 tablet_service.cc:1467] Processing CreateTablet for tablet 0d85098d049b4a85ab0f36af6d7a383e (DEFAULT_TABLE table=rebalance_test_table_2 [id=f0628fbc30d64bd9be053cdd8aaf49f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:06.858744 21470 tablet_service.cc:1467] Processing CreateTablet for tablet 0309e22baa9e492281e2f35196d9ca5f (DEFAULT_TABLE table=rebalance_test_table_2 [id=f0628fbc30d64bd9be053cdd8aaf49f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:06.859105 21471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d85098d049b4a85ab0f36af6d7a383e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:06.859356 21470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0309e22baa9e492281e2f35196d9ca5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:06.859457 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 67d8b1dfeecc48d29976027140ba3656 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f0628fbc30d64bd9be053cdd8aaf49f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:06.859835 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d8b1dfeecc48d29976027140ba3656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:06.864485 21879 tablet_bootstrap.cc:492] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9: Bootstrap starting.
I20240501 20:11:06.865571 21878 tablet_bootstrap.cc:492] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: Bootstrap starting.
I20240501 20:11:06.865695 21879 tablet_bootstrap.cc:654] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:06.867020 21878 tablet_bootstrap.cc:654] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:06.867563 21879 tablet_bootstrap.cc:492] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9: No bootstrap required, opened a new log
I20240501 20:11:06.867697 21879 ts_tablet_manager.cc:1367] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:11:06.868459 21879 raft_consensus.cc:348] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.868643 21879 raft_consensus.cc:374] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:06.868736 21879 raft_consensus.cc:729] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Initialized, Role: FOLLOWER
I20240501 20:11:06.868963 21879 consensus_queue.cc:260] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.869160 21879 raft_consensus.cc:388] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:06.869237 21879 raft_consensus.cc:482] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:06.869341 21879 raft_consensus.cc:3037] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:06.871415 21878 tablet_bootstrap.cc:492] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: No bootstrap required, opened a new log
I20240501 20:11:06.871567 21878 ts_tablet_manager.cc:1367] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240501 20:11:06.871732 21879 raft_consensus.cc:504] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.871989 21879 leader_election.cc:302] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:06.872189 21879 leader_election.cc:288] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:06.872263 21888 raft_consensus.cc:2781] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:06.872278 21878 raft_consensus.cc:348] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.872427 21888 raft_consensus.cc:686] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:06.872439 21878 raft_consensus.cc:374] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:06.872506 21878 raft_consensus.cc:729] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Initialized, Role: FOLLOWER
I20240501 20:11:06.872674 21879 ts_tablet_manager.cc:1398] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.002s
I20240501 20:11:06.872669 21888 consensus_queue.cc:237] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:06.872764 21878 consensus_queue.cc:260] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.872931 21878 raft_consensus.cc:388] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:06.873018 21878 raft_consensus.cc:482] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:06.873081 21878 raft_consensus.cc:3037] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:06.874704 21214 catalog_manager.cc:5484] T 67d8b1dfeecc48d29976027140ba3656 P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.875386 21878 raft_consensus.cc:504] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.875591 21878 leader_election.cc:302] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b80868c0ff4edabaac58ee1c5b41ac; no voters: 
I20240501 20:11:06.875777 21878 leader_election.cc:288] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:06.875880 21883 raft_consensus.cc:2781] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:06.876205 21878 ts_tablet_manager.cc:1398] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 20:11:06.876474 21878 tablet_bootstrap.cc:492] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: Bootstrap starting.
I20240501 20:11:06.876582 21883 raft_consensus.cc:686] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Becoming Leader. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Running, Role: LEADER
I20240501 20:11:06.876823 21883 consensus_queue.cc:237] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.877957 21878 tablet_bootstrap.cc:654] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:06.879798 21878 tablet_bootstrap.cc:492] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: No bootstrap required, opened a new log
I20240501 20:11:06.879920 21878 ts_tablet_manager.cc:1367] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.002s
I20240501 20:11:06.880574 21878 raft_consensus.cc:348] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.880748 21878 raft_consensus.cc:374] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:06.880621 21214 catalog_manager.cc:5484] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194). New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.880834 21878 raft_consensus.cc:729] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Initialized, Role: FOLLOWER
I20240501 20:11:06.881021 21878 consensus_queue.cc:260] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.881202 21878 raft_consensus.cc:388] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:06.881286 21878 raft_consensus.cc:482] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:06.881392 21878 raft_consensus.cc:3037] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:06.883570 21878 raft_consensus.cc:504] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.883849 21878 leader_election.cc:302] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b80868c0ff4edabaac58ee1c5b41ac; no voters: 
I20240501 20:11:06.884024 21878 leader_election.cc:288] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:06.884215 21883 raft_consensus.cc:2781] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:06.884356 21883 raft_consensus.cc:686] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Becoming Leader. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Running, Role: LEADER
I20240501 20:11:06.884552 21883 consensus_queue.cc:237] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:06.886260 21878 ts_tablet_manager.cc:1398] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.002s
I20240501 20:11:06.887379 21214 catalog_manager.cc:5484] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194). New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.197490 21214 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58054:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 20:11:08.205893 21336 tablet_service.cc:1467] Processing CreateTablet for tablet e058f2c85c084eff963a95b69dd68117 (DEFAULT_TABLE table=rebalance_test_table_3 [id=67a3850865df4d6f952e084e5e7fc54a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:08.206143 21470 tablet_service.cc:1467] Processing CreateTablet for tablet a9e97a6d76434a18a16f554732211897 (DEFAULT_TABLE table=rebalance_test_table_3 [id=67a3850865df4d6f952e084e5e7fc54a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:08.206362 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e058f2c85c084eff963a95b69dd68117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:08.206430 21335 tablet_service.cc:1467] Processing CreateTablet for tablet 6142bb039cc0454b8b2b691fece15f52 (DEFAULT_TABLE table=rebalance_test_table_3 [id=67a3850865df4d6f952e084e5e7fc54a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:08.206597 21470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e97a6d76434a18a16f554732211897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:08.206790 21335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6142bb039cc0454b8b2b691fece15f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:08.211282 21949 tablet_bootstrap.cc:492] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9: Bootstrap starting.
I20240501 20:11:08.212723 21949 tablet_bootstrap.cc:654] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:08.212833 21950 tablet_bootstrap.cc:492] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac: Bootstrap starting.
I20240501 20:11:08.214128 21950 tablet_bootstrap.cc:654] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:08.214413 21949 tablet_bootstrap.cc:492] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9: No bootstrap required, opened a new log
I20240501 20:11:08.214531 21949 ts_tablet_manager.cc:1367] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:11:08.215173 21949 raft_consensus.cc:348] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:08.215340 21949 raft_consensus.cc:374] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:08.215430 21949 raft_consensus.cc:729] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Initialized, Role: FOLLOWER
I20240501 20:11:08.215670 21949 consensus_queue.cc:260] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:08.215845 21949 raft_consensus.cc:388] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:08.215870 21950 tablet_bootstrap.cc:492] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac: No bootstrap required, opened a new log
I20240501 20:11:08.215925 21949 raft_consensus.cc:482] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:08.215998 21950 ts_tablet_manager.cc:1367] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:11:08.216027 21949 raft_consensus.cc:3037] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:08.216714 21950 raft_consensus.cc:348] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:08.216840 21950 raft_consensus.cc:374] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:08.216917 21950 raft_consensus.cc:729] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Initialized, Role: FOLLOWER
I20240501 20:11:08.217099 21950 consensus_queue.cc:260] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:08.217231 21950 raft_consensus.cc:388] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:08.217299 21950 raft_consensus.cc:482] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:08.217393 21950 raft_consensus.cc:3037] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:08.217917 21949 raft_consensus.cc:504] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:08.218122 21949 leader_election.cc:302] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:08.218529 21949 leader_election.cc:288] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:08.218672 21953 raft_consensus.cc:2781] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:08.219432 21950 raft_consensus.cc:504] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:08.219645 21950 leader_election.cc:302] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b80868c0ff4edabaac58ee1c5b41ac; no voters: 
I20240501 20:11:08.219074 21953 raft_consensus.cc:686] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:08.219974 21953 consensus_queue.cc:237] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:08.220719 21950 leader_election.cc:288] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:08.220855 21956 raft_consensus.cc:2781] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:08.225443 21949 ts_tablet_manager.cc:1398] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20240501 20:11:08.225510 21956 raft_consensus.cc:686] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Becoming Leader. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Running, Role: LEADER
I20240501 20:11:08.225769 21949 tablet_bootstrap.cc:492] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: Bootstrap starting.
I20240501 20:11:08.225850 21956 consensus_queue.cc:237] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } }
I20240501 20:11:08.227358 21949 tablet_bootstrap.cc:654] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:08.229053 21214 catalog_manager.cc:5484] T e058f2c85c084eff963a95b69dd68117 P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.231141 21213 catalog_manager.cc:5484] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194). New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.233989 21950 ts_tablet_manager.cc:1398] T a9e97a6d76434a18a16f554732211897 P 83b80868c0ff4edabaac58ee1c5b41ac: Time spent starting tablet: real 0.018s	user 0.010s	sys 0.008s
I20240501 20:11:08.235862 21949 tablet_bootstrap.cc:492] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: No bootstrap required, opened a new log
I20240501 20:11:08.236056 21949 ts_tablet_manager.cc:1367] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.004s
I20240501 20:11:08.236816 21949 raft_consensus.cc:348] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:08.237020 21949 raft_consensus.cc:374] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:08.237635 21949 raft_consensus.cc:729] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Initialized, Role: FOLLOWER
I20240501 20:11:08.237887 21949 consensus_queue.cc:260] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:08.238051 21949 raft_consensus.cc:388] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:08.238173 21949 raft_consensus.cc:482] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:08.238273 21949 raft_consensus.cc:3037] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:08.240144 21949 raft_consensus.cc:504] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:08.240316 21949 leader_election.cc:302] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:08.240499 21949 leader_election.cc:288] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:08.240554 21953 raft_consensus.cc:2781] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:08.240715 21953 raft_consensus.cc:686] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:08.240957 21949 ts_tablet_manager.cc:1398] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 20:11:08.241163 21953 consensus_queue.cc:237] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } }
I20240501 20:11:08.242897 21214 catalog_manager.cc:5484] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.710791  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:34785
--local_ip_for_outbound_sockets=127.8.68.195
--tserver_master_addrs=127.8.68.254:43769
--webserver_port=38371
--webserver_interface=127.8.68.195
--builtin_ntp_servers=127.8.68.212:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:09.033694 21986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:09.033946 21986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:09.034088 21986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:09.041384 21986 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:09.041500 21986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:09.041637 21986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240501 20:11:09.050014 21986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:34785
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=38371
--tserver_master_addrs=127.8.68.254:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:09.051287 21986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:09.052892 21986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:09.064505 21991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:09.065026 21992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:09.065459 21994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:09.066056 21986 server_base.cc:1034] running on GCE node
I20240501 20:11:09.066601 21986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:09.067473 21986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:09.068660 21986 hybrid_clock.cc:648] HybridClock initialized: now 1714594269068621 us; error 30 us; skew 500 ppm
I20240501 20:11:09.068933 21986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:09.070930 21986 webserver.cc:460] Webserver started at http://127.8.68.195:38371/ using document root <none> and password file <none>
I20240501 20:11:09.071579 21986 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:09.071700 21986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:09.076334 21986 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 20:11:09.078828 22000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:09.079977 21986 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:09.080116 21986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "12258cbda618428d9089acc34e1565d3"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-08p9"
I20240501 20:11:09.080833 21986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:09.110693 21986 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:09.111353 21986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:09.111598 21986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:09.112393 21986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:09.113689 21986 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:09.113762 21986 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:09.113880 21986 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:09.113931 21986 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:09.141742 21986 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:34785
I20240501 20:11:09.141793 22112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:34785 every 8 connection(s)
I20240501 20:11:09.143049 21986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:11:09.148312  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 21986
I20240501 20:11:09.149021  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:44331
--local_ip_for_outbound_sockets=127.8.68.196
--tserver_master_addrs=127.8.68.254:43769
--webserver_port=44839
--webserver_interface=127.8.68.196
--builtin_ntp_servers=127.8.68.212:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:09.154649 22113 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43769
I20240501 20:11:09.154896 22113 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:09.155597 22113 heartbeater.cc:511] Master 127.8.68.254:43769 requested a full tablet report, sending...
I20240501 20:11:09.157197 21214 ts_manager.cc:194] Re-registered known tserver with Master: 12258cbda618428d9089acc34e1565d3 (127.8.68.195:34785)
I20240501 20:11:09.158150 21214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:39735
W20240501 20:11:09.466854 22117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:09.467090 22117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:09.467200 22117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:09.474172 22117 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:09.474267 22117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:09.474431 22117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:11:09.482807 22117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:44331
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=44839
--tserver_master_addrs=127.8.68.254:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:09.483850 22117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:09.485466 22117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:09.496644 22123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:09.497051 22122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:09.497583 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:09.498104 22117 server_base.cc:1034] running on GCE node
I20240501 20:11:09.498654 22117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:09.499445 22117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:09.500640 22117 hybrid_clock.cc:648] HybridClock initialized: now 1714594269500606 us; error 43 us; skew 500 ppm
I20240501 20:11:09.500947 22117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:09.503196 22117 webserver.cc:460] Webserver started at http://127.8.68.196:44839/ using document root <none> and password file <none>
I20240501 20:11:09.503827 22117 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:09.503947 22117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:09.508651 22117 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 20:11:09.511102 22131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:09.512269 22117 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:09.512413 22117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f5235e7de11c49d8b85ebeb503eeaeae"
format_stamp: "Formatted at 2024-05-01 20:11:00 on dist-test-slave-08p9"
I20240501 20:11:09.513131 22117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:09.525779 22117 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:09.526465 22117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:09.526722 22117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:09.527513 22117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:09.528875 22117 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:09.528950 22117 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:09.529062 22117 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:09.529119 22117 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:09.557322 22117 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:44331
I20240501 20:11:09.557407 22243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:44331 every 8 connection(s)
I20240501 20:11:09.558612 22117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:09.566409  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 22117
I20240501 20:11:09.570780 22244 heartbeater.cc:347] Connected to a master server at 127.8.68.254:43769
I20240501 20:11:09.571022 22244 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:09.571828 22244 heartbeater.cc:511] Master 127.8.68.254:43769 requested a full tablet report, sending...
I20240501 20:11:09.573072 21214 ts_manager.cc:194] Re-registered known tserver with Master: f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196:44331)
I20240501 20:11:09.573818 21214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:45875
I20240501 20:11:10.014006 22048 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.015092 21470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:10.018822 21335 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:10.036851 22179 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.160307 22113 heartbeater.cc:503] Master 127.8.68.254:43769 was elected leader, sending a full tablet report...
I20240501 20:11:10.189518 22048 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.194155 21470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:10.197420 22179 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.200831 21335 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:10.363965 22048 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.370055 21470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:10.374969 22179 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.387383 21335 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:10.447649 21491 consensus_queue.cc:237] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } }
I20240501 20:11:10.450074 22359 raft_consensus.cc:2932] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 12258cbda618428d9089acc34e1565d3 (127.8.68.195) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } } }
I20240501 20:11:10.455722 21211 catalog_manager.cc:5484] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: config changed from index -1 to 3, NON_VOTER 12258cbda618428d9089acc34e1565d3 (127.8.68.195) added. New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:10.458149 21423 consensus_peers.cc:485] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac -> Peer 12258cbda618428d9089acc34e1565d3 (127.8.68.195:34785): Couldn't send request to peer 12258cbda618428d9089acc34e1565d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d85098d049b4a85ab0f36af6d7a383e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:10.467880 21356 consensus_queue.cc:237] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } }
I20240501 20:11:10.470372 22373 raft_consensus.cc:2932] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } } }
I20240501 20:11:10.475910 21211 catalog_manager.cc:5484] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: config changed from index -1 to 2, NON_VOTER f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:10.480109 21291 consensus_peers.cc:485] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 -> Peer f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196:44331): Couldn't send request to peer f5235e7de11c49d8b85ebeb503eeaeae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6142bb039cc0454b8b2b691fece15f52. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:10.486423 21491 consensus_queue.cc:237] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } }
I20240501 20:11:10.487875 22359 raft_consensus.cc:2932] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 12258cbda618428d9089acc34e1565d3 (127.8.68.195) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } } }
W20240501 20:11:10.488281 21423 consensus_peers.cc:485] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac -> Peer 12258cbda618428d9089acc34e1565d3 (127.8.68.195:34785): Couldn't send request to peer 12258cbda618428d9089acc34e1565d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fad1c1fd383d45dba455731bebfcbbf5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:10.491830 21211 catalog_manager.cc:5484] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: config changed from index -1 to 2, NON_VOTER 12258cbda618428d9089acc34e1565d3 (127.8.68.195) added. New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:10.502688 21356 consensus_queue.cc:237] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } }
I20240501 20:11:10.504009 22373 raft_consensus.cc:2932] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } } }
W20240501 20:11:10.504459 21291 consensus_peers.cc:485] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 -> Peer f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196:44331): Couldn't send request to peer f5235e7de11c49d8b85ebeb503eeaeae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6cf57ad231d4706a4a0b62845d90b89. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:10.508328 21211 catalog_manager.cc:5484] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: config changed from index -1 to 2, NON_VOTER f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:10.516940 21491 consensus_queue.cc:237] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } }
I20240501 20:11:10.518400 22381 raft_consensus.cc:2932] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } } }
I20240501 20:11:10.523197 21211 catalog_manager.cc:5484] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: config changed from index -1 to 2, NON_VOTER f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:10.525950 21425 consensus_peers.cc:485] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac -> Peer f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196:44331): Couldn't send request to peer f5235e7de11c49d8b85ebeb503eeaeae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0309e22baa9e492281e2f35196d9ca5f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:10.533296 21356 consensus_queue.cc:237] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } }
I20240501 20:11:10.534986 22372 raft_consensus.cc:2932] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 12258cbda618428d9089acc34e1565d3 (127.8.68.195) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } } }
I20240501 20:11:10.539980 21211 catalog_manager.cc:5484] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: config changed from index -1 to 2, NON_VOTER 12258cbda618428d9089acc34e1565d3 (127.8.68.195) added. New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:10.541923 21289 consensus_peers.cc:485] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 -> Peer 12258cbda618428d9089acc34e1565d3 (127.8.68.195:34785): Couldn't send request to peer 12258cbda618428d9089acc34e1565d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 837412d5b37c4b1791f56e6248283aad. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:10.576134 22244 heartbeater.cc:503] Master 127.8.68.254:43769 was elected leader, sending a full tablet report...
I20240501 20:11:10.886597 22460 ts_tablet_manager.cc:902] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: Initiating tablet copy from peer 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847)
I20240501 20:11:10.888207 22460 tablet_copy_client.cc:308] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:38847
I20240501 20:11:10.894140 21511 tablet_copy_service.cc:140] P 83b80868c0ff4edabaac58ee1c5b41ac: Received BeginTabletCopySession request for tablet fad1c1fd383d45dba455731bebfcbbf5 from peer 12258cbda618428d9089acc34e1565d3 ({username='slave'} at 127.8.68.195:58315)
I20240501 20:11:10.894388 21511 tablet_copy_service.cc:161] P 83b80868c0ff4edabaac58ee1c5b41ac: Beginning new tablet copy session on tablet fad1c1fd383d45dba455731bebfcbbf5 from peer 12258cbda618428d9089acc34e1565d3 at {username='slave'} at 127.8.68.195:58315: session id = 12258cbda618428d9089acc34e1565d3-fad1c1fd383d45dba455731bebfcbbf5
I20240501 20:11:10.897095 21511 tablet_copy_source_session.cc:205] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:10.898417 22466 ts_tablet_manager.cc:902] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: Initiating tablet copy from peer f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333)
I20240501 20:11:10.900118 22466 tablet_copy_client.cc:308] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:42333
I20240501 20:11:10.902437 22460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad1c1fd383d45dba455731bebfcbbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:10.906925 21376 tablet_copy_service.cc:140] P f6d2ed84d9904640bd42888b0c5779c9: Received BeginTabletCopySession request for tablet 6142bb039cc0454b8b2b691fece15f52 from peer f5235e7de11c49d8b85ebeb503eeaeae ({username='slave'} at 127.8.68.196:46863)
I20240501 20:11:10.907166 21376 tablet_copy_service.cc:161] P f6d2ed84d9904640bd42888b0c5779c9: Beginning new tablet copy session on tablet 6142bb039cc0454b8b2b691fece15f52 from peer f5235e7de11c49d8b85ebeb503eeaeae at {username='slave'} at 127.8.68.196:46863: session id = f5235e7de11c49d8b85ebeb503eeaeae-6142bb039cc0454b8b2b691fece15f52
I20240501 20:11:10.910494 21376 tablet_copy_source_session.cc:205] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:10.914045 22460 tablet_copy_client.cc:783] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:10.914422 22460 tablet_copy_client.cc:647] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:10.914947 22466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6142bb039cc0454b8b2b691fece15f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:10.916649 22460 tablet_copy_client.cc:523] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:10.920529 22460 tablet_bootstrap.cc:492] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: Bootstrap starting.
I20240501 20:11:10.927249 22466 tablet_copy_client.cc:783] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:10.927614 22466 tablet_copy_client.cc:647] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:10.928066 22460 log.cc:824] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:10.929746 22466 tablet_copy_client.cc:523] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:10.931650 22460 tablet_bootstrap.cc:492] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:10.932101 22460 tablet_bootstrap.cc:492] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: Bootstrap complete.
I20240501 20:11:10.932503 22460 ts_tablet_manager.cc:1367] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: Time spent bootstrapping tablet: real 0.012s	user 0.011s	sys 0.001s
I20240501 20:11:10.933426 22466 tablet_bootstrap.cc:492] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: Bootstrap starting.
I20240501 20:11:10.940526 22466 log.cc:824] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:10.940490 22460 raft_consensus.cc:348] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } }
I20240501 20:11:10.940902 22460 raft_consensus.cc:729] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12258cbda618428d9089acc34e1565d3, State: Initialized, Role: LEARNER
I20240501 20:11:10.941350 22460 consensus_queue.cc:260] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } }
I20240501 20:11:10.943007 22460 ts_tablet_manager.cc:1398] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.004s
I20240501 20:11:10.943890 21511 tablet_copy_service.cc:342] P 83b80868c0ff4edabaac58ee1c5b41ac: Request end of tablet copy session 12258cbda618428d9089acc34e1565d3-fad1c1fd383d45dba455731bebfcbbf5 received from {username='slave'} at 127.8.68.195:58315
I20240501 20:11:10.944038 21511 tablet_copy_service.cc:433] P 83b80868c0ff4edabaac58ee1c5b41ac: ending tablet copy session 12258cbda618428d9089acc34e1565d3-fad1c1fd383d45dba455731bebfcbbf5 on tablet fad1c1fd383d45dba455731bebfcbbf5 with peer 12258cbda618428d9089acc34e1565d3
I20240501 20:11:10.944815 22466 tablet_bootstrap.cc:492] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:10.945290 22466 tablet_bootstrap.cc:492] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: Bootstrap complete.
I20240501 20:11:10.945681 22466 ts_tablet_manager.cc:1367] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: Time spent bootstrapping tablet: real 0.012s	user 0.013s	sys 0.000s
I20240501 20:11:10.951161 22466 raft_consensus.cc:348] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } }
I20240501 20:11:10.951472 22466 raft_consensus.cc:729] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5235e7de11c49d8b85ebeb503eeaeae, State: Initialized, Role: LEARNER
I20240501 20:11:10.951855 22466 consensus_queue.cc:260] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } }
I20240501 20:11:10.952982 22466 ts_tablet_manager.cc:1398] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20240501 20:11:10.953780 21376 tablet_copy_service.cc:342] P f6d2ed84d9904640bd42888b0c5779c9: Request end of tablet copy session f5235e7de11c49d8b85ebeb503eeaeae-6142bb039cc0454b8b2b691fece15f52 received from {username='slave'} at 127.8.68.196:46863
I20240501 20:11:10.953922 21376 tablet_copy_service.cc:433] P f6d2ed84d9904640bd42888b0c5779c9: ending tablet copy session f5235e7de11c49d8b85ebeb503eeaeae-6142bb039cc0454b8b2b691fece15f52 on tablet 6142bb039cc0454b8b2b691fece15f52 with peer f5235e7de11c49d8b85ebeb503eeaeae
I20240501 20:11:11.022629 22466 ts_tablet_manager.cc:902] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: Initiating tablet copy from peer 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847)
I20240501 20:11:11.023869 22466 tablet_copy_client.cc:308] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:38847
I20240501 20:11:11.029620 21511 tablet_copy_service.cc:140] P 83b80868c0ff4edabaac58ee1c5b41ac: Received BeginTabletCopySession request for tablet 0309e22baa9e492281e2f35196d9ca5f from peer f5235e7de11c49d8b85ebeb503eeaeae ({username='slave'} at 127.8.68.196:45365)
I20240501 20:11:11.029778 21511 tablet_copy_service.cc:161] P 83b80868c0ff4edabaac58ee1c5b41ac: Beginning new tablet copy session on tablet 0309e22baa9e492281e2f35196d9ca5f from peer f5235e7de11c49d8b85ebeb503eeaeae at {username='slave'} at 127.8.68.196:45365: session id = f5235e7de11c49d8b85ebeb503eeaeae-0309e22baa9e492281e2f35196d9ca5f
I20240501 20:11:11.031862 21511 tablet_copy_source_session.cc:205] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:11.032713 22460 ts_tablet_manager.cc:902] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: Initiating tablet copy from peer 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847)
I20240501 20:11:11.033004 22466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0309e22baa9e492281e2f35196d9ca5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:11.034940 22460 tablet_copy_client.cc:308] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:38847
I20240501 20:11:11.035645 21511 tablet_copy_service.cc:140] P 83b80868c0ff4edabaac58ee1c5b41ac: Received BeginTabletCopySession request for tablet 0d85098d049b4a85ab0f36af6d7a383e from peer 12258cbda618428d9089acc34e1565d3 ({username='slave'} at 127.8.68.195:58315)
I20240501 20:11:11.035775 21511 tablet_copy_service.cc:161] P 83b80868c0ff4edabaac58ee1c5b41ac: Beginning new tablet copy session on tablet 0d85098d049b4a85ab0f36af6d7a383e from peer 12258cbda618428d9089acc34e1565d3 at {username='slave'} at 127.8.68.195:58315: session id = 12258cbda618428d9089acc34e1565d3-0d85098d049b4a85ab0f36af6d7a383e
I20240501 20:11:11.038162 22466 tablet_copy_client.cc:783] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:11.038424 22466 tablet_copy_client.cc:647] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:11.038662 21511 tablet_copy_source_session.cc:205] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:11.040238 22460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d85098d049b4a85ab0f36af6d7a383e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:11.040243 22466 tablet_copy_client.cc:523] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.043473 22466 tablet_bootstrap.cc:492] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: Bootstrap starting.
I20240501 20:11:11.044662 22460 tablet_copy_client.cc:783] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:11.044886 22460 tablet_copy_client.cc:647] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:11.046249 22460 tablet_copy_client.cc:523] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.048540 22460 tablet_bootstrap.cc:492] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: Bootstrap starting.
I20240501 20:11:11.051280 22485 ts_tablet_manager.cc:902] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: Initiating tablet copy from peer f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333)
I20240501 20:11:11.051971 22466 tablet_bootstrap.cc:492] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.052459 22466 tablet_bootstrap.cc:492] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: Bootstrap complete.
I20240501 20:11:11.052743 22466 ts_tablet_manager.cc:1367] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.004s
I20240501 20:11:11.053400 22485 tablet_copy_client.cc:308] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:42333
I20240501 20:11:11.053637 22466 raft_consensus.cc:348] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } }
I20240501 20:11:11.053916 22466 raft_consensus.cc:729] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5235e7de11c49d8b85ebeb503eeaeae, State: Initialized, Role: LEARNER
I20240501 20:11:11.054167 22466 consensus_queue.cc:260] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } }
I20240501 20:11:11.054725 22466 ts_tablet_manager.cc:1398] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:11:11.055354 21511 tablet_copy_service.cc:342] P 83b80868c0ff4edabaac58ee1c5b41ac: Request end of tablet copy session f5235e7de11c49d8b85ebeb503eeaeae-0309e22baa9e492281e2f35196d9ca5f received from {username='slave'} at 127.8.68.196:45365
I20240501 20:11:11.055476 21511 tablet_copy_service.cc:433] P 83b80868c0ff4edabaac58ee1c5b41ac: ending tablet copy session f5235e7de11c49d8b85ebeb503eeaeae-0309e22baa9e492281e2f35196d9ca5f on tablet 0309e22baa9e492281e2f35196d9ca5f with peer f5235e7de11c49d8b85ebeb503eeaeae
I20240501 20:11:11.061044 22460 tablet_bootstrap.cc:492] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.061529 21376 tablet_copy_service.cc:140] P f6d2ed84d9904640bd42888b0c5779c9: Received BeginTabletCopySession request for tablet 837412d5b37c4b1791f56e6248283aad from peer 12258cbda618428d9089acc34e1565d3 ({username='slave'} at 127.8.68.195:34255)
I20240501 20:11:11.061681 21376 tablet_copy_service.cc:161] P f6d2ed84d9904640bd42888b0c5779c9: Beginning new tablet copy session on tablet 837412d5b37c4b1791f56e6248283aad from peer 12258cbda618428d9089acc34e1565d3 at {username='slave'} at 127.8.68.195:34255: session id = 12258cbda618428d9089acc34e1565d3-837412d5b37c4b1791f56e6248283aad
I20240501 20:11:11.063588 21376 tablet_copy_source_session.cc:205] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:11.064100 22460 tablet_bootstrap.cc:492] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: Bootstrap complete.
I20240501 20:11:11.064394 22460 ts_tablet_manager.cc:1367] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.001s
I20240501 20:11:11.064596 22485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837412d5b37c4b1791f56e6248283aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:11.065218 22460 raft_consensus.cc:348] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } }
I20240501 20:11:11.065431 22460 raft_consensus.cc:729] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12258cbda618428d9089acc34e1565d3, State: Initialized, Role: LEARNER
I20240501 20:11:11.065626 22460 consensus_queue.cc:260] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } }
I20240501 20:11:11.066231 22460 ts_tablet_manager.cc:1398] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:11:11.066988 21511 tablet_copy_service.cc:342] P 83b80868c0ff4edabaac58ee1c5b41ac: Request end of tablet copy session 12258cbda618428d9089acc34e1565d3-0d85098d049b4a85ab0f36af6d7a383e received from {username='slave'} at 127.8.68.195:58315
I20240501 20:11:11.067121 21511 tablet_copy_service.cc:433] P 83b80868c0ff4edabaac58ee1c5b41ac: ending tablet copy session 12258cbda618428d9089acc34e1565d3-0d85098d049b4a85ab0f36af6d7a383e on tablet 0d85098d049b4a85ab0f36af6d7a383e with peer 12258cbda618428d9089acc34e1565d3
I20240501 20:11:11.070822 22485 tablet_copy_client.cc:783] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:11.071038 22485 tablet_copy_client.cc:647] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:11.072408 22485 tablet_copy_client.cc:523] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.075258 22485 tablet_bootstrap.cc:492] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: Bootstrap starting.
I20240501 20:11:11.080318 22485 tablet_bootstrap.cc:492] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.080628 22485 tablet_bootstrap.cc:492] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: Bootstrap complete.
I20240501 20:11:11.080811 22485 ts_tablet_manager.cc:1367] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 20:11:11.081419 22485 raft_consensus.cc:348] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } }
I20240501 20:11:11.081560 22485 raft_consensus.cc:729] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12258cbda618428d9089acc34e1565d3, State: Initialized, Role: LEARNER
I20240501 20:11:11.081739 22485 consensus_queue.cc:260] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: NON_VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: true } }
I20240501 20:11:11.082315 22485 ts_tablet_manager.cc:1398] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3: Time spent starting tablet: real 0.001s	user 0.003s	sys 0.000s
I20240501 20:11:11.083976 21376 tablet_copy_service.cc:342] P f6d2ed84d9904640bd42888b0c5779c9: Request end of tablet copy session 12258cbda618428d9089acc34e1565d3-837412d5b37c4b1791f56e6248283aad received from {username='slave'} at 127.8.68.195:34255
I20240501 20:11:11.084120 21376 tablet_copy_service.cc:433] P f6d2ed84d9904640bd42888b0c5779c9: ending tablet copy session 12258cbda618428d9089acc34e1565d3-837412d5b37c4b1791f56e6248283aad on tablet 837412d5b37c4b1791f56e6248283aad with peer 12258cbda618428d9089acc34e1565d3
I20240501 20:11:11.096026 22466 ts_tablet_manager.cc:902] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: Initiating tablet copy from peer f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333)
I20240501 20:11:11.097133 22466 tablet_copy_client.cc:308] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:42333
I20240501 20:11:11.097754 21376 tablet_copy_service.cc:140] P f6d2ed84d9904640bd42888b0c5779c9: Received BeginTabletCopySession request for tablet f6cf57ad231d4706a4a0b62845d90b89 from peer f5235e7de11c49d8b85ebeb503eeaeae ({username='slave'} at 127.8.68.196:46863)
I20240501 20:11:11.097878 21376 tablet_copy_service.cc:161] P f6d2ed84d9904640bd42888b0c5779c9: Beginning new tablet copy session on tablet f6cf57ad231d4706a4a0b62845d90b89 from peer f5235e7de11c49d8b85ebeb503eeaeae at {username='slave'} at 127.8.68.196:46863: session id = f5235e7de11c49d8b85ebeb503eeaeae-f6cf57ad231d4706a4a0b62845d90b89
I20240501 20:11:11.100179 21376 tablet_copy_source_session.cc:205] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:11.101241 22466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cf57ad231d4706a4a0b62845d90b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:11.106741 22466 tablet_copy_client.cc:783] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:11.106992 22466 tablet_copy_client.cc:647] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:11.108409 22466 tablet_copy_client.cc:523] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.110688 22466 tablet_bootstrap.cc:492] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: Bootstrap starting.
I20240501 20:11:11.121902 22466 tablet_bootstrap.cc:492] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.122196 22466 tablet_bootstrap.cc:492] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: Bootstrap complete.
I20240501 20:11:11.122432 22466 ts_tablet_manager.cc:1367] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: Time spent bootstrapping tablet: real 0.012s	user 0.011s	sys 0.000s
I20240501 20:11:11.123090 22466 raft_consensus.cc:348] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } }
I20240501 20:11:11.123320 22466 raft_consensus.cc:729] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5235e7de11c49d8b85ebeb503eeaeae, State: Initialized, Role: LEARNER
I20240501 20:11:11.123538 22466 consensus_queue.cc:260] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: true } }
I20240501 20:11:11.123967 22466 ts_tablet_manager.cc:1398] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240501 20:11:11.124634 21376 tablet_copy_service.cc:342] P f6d2ed84d9904640bd42888b0c5779c9: Request end of tablet copy session f5235e7de11c49d8b85ebeb503eeaeae-f6cf57ad231d4706a4a0b62845d90b89 received from {username='slave'} at 127.8.68.196:46863
I20240501 20:11:11.124799 21376 tablet_copy_service.cc:433] P f6d2ed84d9904640bd42888b0c5779c9: ending tablet copy session f5235e7de11c49d8b85ebeb503eeaeae-f6cf57ad231d4706a4a0b62845d90b89 on tablet f6cf57ad231d4706a4a0b62845d90b89 with peer f5235e7de11c49d8b85ebeb503eeaeae
I20240501 20:11:11.409044 22068 raft_consensus.cc:1203] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240501 20:11:11.429704 22199 raft_consensus.cc:1203] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 20:11:11.431372 22068 raft_consensus.cc:1203] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 20:11:11.453934 22068 raft_consensus.cc:1203] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 20:11:11.500943 22199 raft_consensus.cc:1203] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 20:11:11.635634 22199 raft_consensus.cc:1203] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 20:11:11.848750 22563 raft_consensus.cc:1050] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: attempting to promote NON_VOTER 12258cbda618428d9089acc34e1565d3 to VOTER
I20240501 20:11:11.849570 22563 consensus_queue.cc:237] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:11.851828 22068 raft_consensus.cc:1261] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Refusing update from remote peer 83b80868c0ff4edabaac58ee1c5b41ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:11.852483 22381 consensus_queue.cc:1035] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:11.861856 22563 raft_consensus.cc:2932] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 12258cbda618428d9089acc34e1565d3 (127.8.68.195) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } } }
I20240501 20:11:11.863381 22068 raft_consensus.cc:2932] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 12258cbda618428d9089acc34e1565d3 (127.8.68.195) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } } }
I20240501 20:11:11.866261 21211 catalog_manager.cc:5484] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: config changed from index 3 to 4, 12258cbda618428d9089acc34e1565d3 (127.8.68.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:11.889178 21491 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d85098d049b4a85ab0f36af6d7a383e"
dest_uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34190
I20240501 20:11:11.889376 21491 raft_consensus.cc:595] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:11.907308 22565 raft_consensus.cc:1050] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: attempting to promote NON_VOTER 12258cbda618428d9089acc34e1565d3 to VOTER
I20240501 20:11:11.907999 22565 consensus_queue.cc:237] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:11.909631 22067 raft_consensus.cc:1261] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Refusing update from remote peer f6d2ed84d9904640bd42888b0c5779c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:11.910005 22381 raft_consensus.cc:1050] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: attempting to promote NON_VOTER 12258cbda618428d9089acc34e1565d3 to VOTER
I20240501 20:11:11.910199 22566 consensus_queue.cc:1035] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:11.910543 22381 consensus_queue.cc:237] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:11.912927 22068 raft_consensus.cc:1261] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 1 LEARNER]: Refusing update from remote peer 83b80868c0ff4edabaac58ee1c5b41ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:11.913409 22563 consensus_queue.cc:1035] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:11.913882 22565 raft_consensus.cc:2932] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 12258cbda618428d9089acc34e1565d3 (127.8.68.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } } }
I20240501 20:11:11.914534 22068 raft_consensus.cc:2932] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 12258cbda618428d9089acc34e1565d3 (127.8.68.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } } }
I20240501 20:11:11.916043 22381 raft_consensus.cc:2932] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 12258cbda618428d9089acc34e1565d3 (127.8.68.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } } }
I20240501 20:11:11.916676 22067 raft_consensus.cc:2932] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 12258cbda618428d9089acc34e1565d3 (127.8.68.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } } }
I20240501 20:11:11.918498 21210 catalog_manager.cc:5484] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: config changed from index 2 to 3, 12258cbda618428d9089acc34e1565d3 (127.8.68.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:11.920737 21211 catalog_manager.cc:5484] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: config changed from index 2 to 3, 12258cbda618428d9089acc34e1565d3 (127.8.68.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:11.981237 22566 raft_consensus.cc:1050] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: attempting to promote NON_VOTER f5235e7de11c49d8b85ebeb503eeaeae to VOTER
I20240501 20:11:11.981796 22566 consensus_queue.cc:237] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:11.983633 22199 raft_consensus.cc:1261] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Refusing update from remote peer f6d2ed84d9904640bd42888b0c5779c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:11.984138 22565 consensus_queue.cc:1035] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:11.987243 22566 raft_consensus.cc:2932] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } } }
I20240501 20:11:11.987978 22199 raft_consensus.cc:2932] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } } }
I20240501 20:11:11.992097 21210 catalog_manager.cc:5484] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: config changed from index 2 to 3, f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.122712 22566 raft_consensus.cc:1050] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: attempting to promote NON_VOTER f5235e7de11c49d8b85ebeb503eeaeae to VOTER
I20240501 20:11:12.123353 22566 consensus_queue.cc:237] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.124995 22199 raft_consensus.cc:1261] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Refusing update from remote peer f6d2ed84d9904640bd42888b0c5779c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:12.125478 22565 consensus_queue.cc:1035] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:12.128042 22566 raft_consensus.cc:2932] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } } }
I20240501 20:11:12.128680 22199 raft_consensus.cc:2932] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } } }
I20240501 20:11:12.132716 21211 catalog_manager.cc:5484] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 reported cstate change: config changed from index 2 to 3, f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6d2ed84d9904640bd42888b0c5779c9" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.134829 21356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "837412d5b37c4b1791f56e6248283aad"
dest_uuid: "f6d2ed84d9904640bd42888b0c5779c9"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49656
I20240501 20:11:12.135080 21356 raft_consensus.cc:595] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.152585 21491 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fad1c1fd383d45dba455731bebfcbbf5"
dest_uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34230
I20240501 20:11:12.152774 21491 raft_consensus.cc:595] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.168578 21491 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d85098d049b4a85ab0f36af6d7a383e"
dest_uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34252
I20240501 20:11:12.168748 21491 raft_consensus.cc:595] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.184927 21356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6142bb039cc0454b8b2b691fece15f52"
dest_uuid: "f6d2ed84d9904640bd42888b0c5779c9"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49670
I20240501 20:11:12.185106 21356 raft_consensus.cc:595] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.215411 22564 raft_consensus.cc:1050] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: attempting to promote NON_VOTER f5235e7de11c49d8b85ebeb503eeaeae to VOTER
I20240501 20:11:12.215951 22564 consensus_queue.cc:237] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.217772 22199 raft_consensus.cc:1261] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 1 LEARNER]: Refusing update from remote peer 83b80868c0ff4edabaac58ee1c5b41ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:11:12.218273 22381 consensus_queue.cc:1035] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:11:12.220810 22564 raft_consensus.cc:2932] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } } }
I20240501 20:11:12.221433 22199 raft_consensus.cc:2932] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } } }
I20240501 20:11:12.224711 21210 catalog_manager.cc:5484] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac reported cstate change: config changed from index 2 to 3, f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.353518 22563 raft_consensus.cc:979] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: : Instructing follower 12258cbda618428d9089acc34e1565d3 to start an election
I20240501 20:11:12.353686 22564 raft_consensus.cc:1067] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Signalling peer 12258cbda618428d9089acc34e1565d3 to start an election
I20240501 20:11:12.354526 22067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fad1c1fd383d45dba455731bebfcbbf5"
dest_uuid: "12258cbda618428d9089acc34e1565d3"
 from {username='slave'} at 127.8.68.194:43095
I20240501 20:11:12.354806 22067 raft_consensus.cc:482] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:12.354990 22067 raft_consensus.cc:3037] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.357115 22067 raft_consensus.cc:504] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:12.358321 22067 leader_election.cc:288] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [CANDIDATE]: Term 2 election: Requested vote from peers 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847)
I20240501 20:11:12.358978 21491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad1c1fd383d45dba455731bebfcbbf5" candidate_uuid: "12258cbda618428d9089acc34e1565d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
I20240501 20:11:12.359269 21491 raft_consensus.cc:3032] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:12.359357 21491 raft_consensus.cc:729] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Running, Role: LEADER
I20240501 20:11:12.359640 21491 consensus_queue.cc:260] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:12.359926 21491 raft_consensus.cc:3037] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.361768 21491 raft_consensus.cc:2454] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12258cbda618428d9089acc34e1565d3 in term 2.
I20240501 20:11:12.362337 22004 leader_election.cc:302] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12258cbda618428d9089acc34e1565d3, 83b80868c0ff4edabaac58ee1c5b41ac; no voters: 
I20240501 20:11:12.362675 22474 raft_consensus.cc:2781] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:12.363087 22474 raft_consensus.cc:686] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 2 LEADER]: Becoming Leader. State: Replica: 12258cbda618428d9089acc34e1565d3, State: Running, Role: LEADER
I20240501 20:11:12.363497 22474 consensus_queue.cc:237] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:12.366930 21211 catalog_manager.cc:5484] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 reported cstate change: term changed from 1 to 2, leader changed from 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194) to 12258cbda618428d9089acc34e1565d3 (127.8.68.195). New cstate: current_term: 2 leader_uuid: "12258cbda618428d9089acc34e1565d3" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.377017 22372 raft_consensus.cc:979] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: : Instructing follower f5235e7de11c49d8b85ebeb503eeaeae to start an election
I20240501 20:11:12.377179 22372 raft_consensus.cc:1067] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Signalling peer f5235e7de11c49d8b85ebeb503eeaeae to start an election
W20240501 20:11:12.377918 21199 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet fad1c1fd383d45dba455731bebfcbbf5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 12258cbda618428d9089acc34e1565d3 (127.8.68.195:34785) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 20:11:12.378041 22199 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6142bb039cc0454b8b2b691fece15f52"
dest_uuid: "f5235e7de11c49d8b85ebeb503eeaeae"
 from {username='slave'} at 127.8.68.193:52963
I20240501 20:11:12.378393 22199 raft_consensus.cc:482] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:12.378561 22199 raft_consensus.cc:3037] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.380625 22199 raft_consensus.cc:504] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.381821 22199 leader_election.cc:288] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [CANDIDATE]: Term 2 election: Requested vote from peers f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333)
I20240501 20:11:12.382496 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6142bb039cc0454b8b2b691fece15f52" candidate_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f6d2ed84d9904640bd42888b0c5779c9"
I20240501 20:11:12.382800 21356 raft_consensus.cc:3032] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:12.382879 21356 raft_consensus.cc:729] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:12.383149 21356 consensus_queue.cc:260] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.383432 21356 raft_consensus.cc:3037] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.385422 21356 raft_consensus.cc:2454] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5235e7de11c49d8b85ebeb503eeaeae in term 2.
I20240501 20:11:12.386672 22135 leader_election.cc:302] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f5235e7de11c49d8b85ebeb503eeaeae, f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:12.387653 22475 raft_consensus.cc:2781] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:12.388562 22475 raft_consensus.cc:686] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 2 LEADER]: Becoming Leader. State: Replica: f5235e7de11c49d8b85ebeb503eeaeae, State: Running, Role: LEADER
I20240501 20:11:12.389050 22475 consensus_queue.cc:237] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.392480 21211 catalog_manager.cc:5484] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae reported cstate change: term changed from 1 to 2, leader changed from f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193) to f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196). New cstate: current_term: 2 leader_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.406534 21491 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0309e22baa9e492281e2f35196d9ca5f"
dest_uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34260
I20240501 20:11:12.406728 21491 raft_consensus.cc:595] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.423206 21356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6cf57ad231d4706a4a0b62845d90b89"
dest_uuid: "f6d2ed84d9904640bd42888b0c5779c9"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49674
I20240501 20:11:12.423382 21356 raft_consensus.cc:595] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.438925 21356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "837412d5b37c4b1791f56e6248283aad"
dest_uuid: "f6d2ed84d9904640bd42888b0c5779c9"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49696
I20240501 20:11:12.439122 21356 raft_consensus.cc:595] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.443111 22563 raft_consensus.cc:979] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: : Instructing follower 12258cbda618428d9089acc34e1565d3 to start an election
I20240501 20:11:12.443280 22564 raft_consensus.cc:1067] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Signalling peer 12258cbda618428d9089acc34e1565d3 to start an election
I20240501 20:11:12.443893 22067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d85098d049b4a85ab0f36af6d7a383e"
dest_uuid: "12258cbda618428d9089acc34e1565d3"
 from {username='slave'} at 127.8.68.194:43095
I20240501 20:11:12.444087 22067 raft_consensus.cc:482] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:12.444213 22067 raft_consensus.cc:3037] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.446919 22067 raft_consensus.cc:504] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:12.447436 22067 leader_election.cc:288] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [CANDIDATE]: Term 2 election: Requested vote from peers 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847)
I20240501 20:11:12.447908 21491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d85098d049b4a85ab0f36af6d7a383e" candidate_uuid: "12258cbda618428d9089acc34e1565d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
I20240501 20:11:12.448087 21491 raft_consensus.cc:3032] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:12.448169 21491 raft_consensus.cc:729] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Running, Role: LEADER
I20240501 20:11:12.448442 21491 consensus_queue.cc:260] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:12.448760 21491 raft_consensus.cc:3037] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.451053 21491 raft_consensus.cc:2454] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12258cbda618428d9089acc34e1565d3 in term 2.
I20240501 20:11:12.451478 22004 leader_election.cc:302] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12258cbda618428d9089acc34e1565d3, 83b80868c0ff4edabaac58ee1c5b41ac; no voters: 
I20240501 20:11:12.451740 22474 raft_consensus.cc:2781] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:12.451900 22474 raft_consensus.cc:686] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 2 LEADER]: Becoming Leader. State: Replica: 12258cbda618428d9089acc34e1565d3, State: Running, Role: LEADER
I20240501 20:11:12.452183 22474 consensus_queue.cc:237] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:12.454226 21211 catalog_manager.cc:5484] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 reported cstate change: term changed from 1 to 2, leader changed from 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194) to 12258cbda618428d9089acc34e1565d3 (127.8.68.195). New cstate: current_term: 2 leader_uuid: "12258cbda618428d9089acc34e1565d3" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.507387 22565 raft_consensus.cc:979] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: : Instructing follower 12258cbda618428d9089acc34e1565d3 to start an election
I20240501 20:11:12.507551 22565 raft_consensus.cc:1067] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Signalling peer 12258cbda618428d9089acc34e1565d3 to start an election
I20240501 20:11:12.508121 22067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "837412d5b37c4b1791f56e6248283aad"
dest_uuid: "12258cbda618428d9089acc34e1565d3"
 from {username='slave'} at 127.8.68.193:46013
I20240501 20:11:12.508283 22067 raft_consensus.cc:482] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:12.508416 22067 raft_consensus.cc:3037] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.510254 22067 raft_consensus.cc:504] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:12.510813 22067 leader_election.cc:288] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [CANDIDATE]: Term 2 election: Requested vote from peers f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333)
I20240501 20:11:12.511289 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837412d5b37c4b1791f56e6248283aad" candidate_uuid: "12258cbda618428d9089acc34e1565d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f6d2ed84d9904640bd42888b0c5779c9"
I20240501 20:11:12.511482 21356 raft_consensus.cc:3032] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:12.511575 21356 raft_consensus.cc:729] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:12.511814 21356 consensus_queue.cc:260] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:12.512056 21356 raft_consensus.cc:3037] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.513752 21356 raft_consensus.cc:2454] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12258cbda618428d9089acc34e1565d3 in term 2.
I20240501 20:11:12.514083 22004 leader_election.cc:302] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12258cbda618428d9089acc34e1565d3, f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:12.514395 22474 raft_consensus.cc:2781] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:12.514545 22474 raft_consensus.cc:686] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 2 LEADER]: Becoming Leader. State: Replica: 12258cbda618428d9089acc34e1565d3, State: Running, Role: LEADER
I20240501 20:11:12.514791 22474 consensus_queue.cc:237] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:12.516494 21211 catalog_manager.cc:5484] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 reported cstate change: term changed from 1 to 2, leader changed from f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193) to 12258cbda618428d9089acc34e1565d3 (127.8.68.195). New cstate: current_term: 2 leader_uuid: "12258cbda618428d9089acc34e1565d3" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.630796 22372 raft_consensus.cc:979] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: : Instructing follower f5235e7de11c49d8b85ebeb503eeaeae to start an election
I20240501 20:11:12.630977 22372 raft_consensus.cc:1067] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Signalling peer f5235e7de11c49d8b85ebeb503eeaeae to start an election
I20240501 20:11:12.631657 22199 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6cf57ad231d4706a4a0b62845d90b89"
dest_uuid: "f5235e7de11c49d8b85ebeb503eeaeae"
 from {username='slave'} at 127.8.68.193:52963
I20240501 20:11:12.631847 22199 raft_consensus.cc:482] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:12.631985 22199 raft_consensus.cc:3037] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.633870 22199 raft_consensus.cc:504] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.634445 22199 leader_election.cc:288] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [CANDIDATE]: Term 2 election: Requested vote from peers f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333)
I20240501 20:11:12.634874 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cf57ad231d4706a4a0b62845d90b89" candidate_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f6d2ed84d9904640bd42888b0c5779c9"
I20240501 20:11:12.635049 21356 raft_consensus.cc:3032] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:12.635140 21356 raft_consensus.cc:729] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d2ed84d9904640bd42888b0c5779c9, State: Running, Role: LEADER
I20240501 20:11:12.635394 21356 consensus_queue.cc:260] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.635634 21356 raft_consensus.cc:3037] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.637533 21356 raft_consensus.cc:2454] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5235e7de11c49d8b85ebeb503eeaeae in term 2.
I20240501 20:11:12.637915 22135 leader_election.cc:302] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f5235e7de11c49d8b85ebeb503eeaeae, f6d2ed84d9904640bd42888b0c5779c9; no voters: 
I20240501 20:11:12.638229 22475 raft_consensus.cc:2781] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:12.638420 22475 raft_consensus.cc:686] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 2 LEADER]: Becoming Leader. State: Replica: f5235e7de11c49d8b85ebeb503eeaeae, State: Running, Role: LEADER
I20240501 20:11:12.638695 22475 consensus_queue.cc:237] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.640535 21211 catalog_manager.cc:5484] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae reported cstate change: term changed from 1 to 2, leader changed from f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193) to f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196). New cstate: current_term: 2 leader_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.686558 21491 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0309e22baa9e492281e2f35196d9ca5f"
dest_uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34290
I20240501 20:11:12.686771 21491 raft_consensus.cc:595] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.716718 22563 raft_consensus.cc:979] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: : Instructing follower f5235e7de11c49d8b85ebeb503eeaeae to start an election
I20240501 20:11:12.716879 22564 raft_consensus.cc:1067] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Signalling peer f5235e7de11c49d8b85ebeb503eeaeae to start an election
I20240501 20:11:12.717525 22199 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0309e22baa9e492281e2f35196d9ca5f"
dest_uuid: "f5235e7de11c49d8b85ebeb503eeaeae"
 from {username='slave'} at 127.8.68.194:46649
I20240501 20:11:12.717725 22199 raft_consensus.cc:482] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:12.717836 22199 raft_consensus.cc:3037] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.719949 22199 raft_consensus.cc:504] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.720465 22199 leader_election.cc:288] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [CANDIDATE]: Term 2 election: Requested vote from peers 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847)
I20240501 20:11:12.720907 21491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0309e22baa9e492281e2f35196d9ca5f" candidate_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83b80868c0ff4edabaac58ee1c5b41ac"
I20240501 20:11:12.721134 21491 raft_consensus.cc:3032] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:12.721222 21491 raft_consensus.cc:729] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b80868c0ff4edabaac58ee1c5b41ac, State: Running, Role: LEADER
I20240501 20:11:12.721499 21491 consensus_queue.cc:260] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.721830 21491 raft_consensus.cc:3037] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.724038 21491 raft_consensus.cc:2454] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5235e7de11c49d8b85ebeb503eeaeae in term 2.
I20240501 20:11:12.724393 22135 leader_election.cc:302] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 83b80868c0ff4edabaac58ee1c5b41ac, f5235e7de11c49d8b85ebeb503eeaeae; no voters: 
I20240501 20:11:12.724674 22475 raft_consensus.cc:2781] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:12.724821 22475 raft_consensus.cc:686] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 2 LEADER]: Becoming Leader. State: Replica: f5235e7de11c49d8b85ebeb503eeaeae, State: Running, Role: LEADER
I20240501 20:11:12.725067 22475 consensus_queue.cc:237] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:12.727289 21210 catalog_manager.cc:5484] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae reported cstate change: term changed from 1 to 2, leader changed from 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194) to f5235e7de11c49d8b85ebeb503eeaeae (127.8.68.196). New cstate: current_term: 2 leader_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.794631 21356 raft_consensus.cc:1261] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Refusing update from remote peer f5235e7de11c49d8b85ebeb503eeaeae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:12.795322 22475 consensus_queue.cc:1035] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.886849 21491 raft_consensus.cc:1261] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Refusing update from remote peer 12258cbda618428d9089acc34e1565d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:12.887621 22474 consensus_queue.cc:1035] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.967655 21491 raft_consensus.cc:1261] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Refusing update from remote peer 12258cbda618428d9089acc34e1565d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:11:12.968465 22708 consensus_queue.cc:1035] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:13.004343 22067 consensus_queue.cc:237] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:13.005116 22708 raft_consensus.cc:2932] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } } }
I20240501 20:11:13.007689 21199 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fad1c1fd383d45dba455731bebfcbbf5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:11:13.008710 21211 catalog_manager.cc:5484] T fad1c1fd383d45dba455731bebfcbbf5 P 12258cbda618428d9089acc34e1565d3 reported cstate change: config changed from index 3 to 5, VOTER 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "12258cbda618428d9089acc34e1565d3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:13.011847 21470 tablet_service.cc:1514] Processing DeleteTablet for tablet fad1c1fd383d45dba455731bebfcbbf5 with delete_type TABLET_DATA_TOMBSTONED (TS 83b80868c0ff4edabaac58ee1c5b41ac not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35790
I20240501 20:11:13.012629 22829 tablet_replica.cc:330] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: stopping tablet replica
I20240501 20:11:13.012996 22829 raft_consensus.cc:2229] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:13.013223 22829 raft_consensus.cc:2258] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:13.014498 22829 ts_tablet_manager.cc:1875] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:13.019011 22829 ts_tablet_manager.cc:1888] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 20:11:13.019135 22829 log.cc:1196] T fad1c1fd383d45dba455731bebfcbbf5 P 83b80868c0ff4edabaac58ee1c5b41ac: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/fad1c1fd383d45dba455731bebfcbbf5
I20240501 20:11:13.019878 21201 catalog_manager.cc:4830] TS 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847): tablet fad1c1fd383d45dba455731bebfcbbf5 (table rebalance_test_table_0 [id=529a9c62044e41128f995a109fe9882c]) successfully deleted
I20240501 20:11:13.023463 22199 consensus_queue.cc:237] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:13.024276 22711 raft_consensus.cc:2932] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } } }
I20240501 20:11:13.027853 21201 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6142bb039cc0454b8b2b691fece15f52 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:11:13.028972 21211 catalog_manager.cc:5484] T 6142bb039cc0454b8b2b691fece15f52 P f5235e7de11c49d8b85ebeb503eeaeae reported cstate change: config changed from index 3 to 5, VOTER f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:13.031944 21335 tablet_service.cc:1514] Processing DeleteTablet for tablet 6142bb039cc0454b8b2b691fece15f52 with delete_type TABLET_DATA_TOMBSTONED (TS f6d2ed84d9904640bd42888b0c5779c9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33814
I20240501 20:11:13.032667 22830 tablet_replica.cc:330] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: stopping tablet replica
I20240501 20:11:13.033011 22830 raft_consensus.cc:2229] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:13.033234 22830 raft_consensus.cc:2258] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:13.034442 22830 ts_tablet_manager.cc:1875] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:13.038826 22830 ts_tablet_manager.cc:1888] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 20:11:13.038944 22830 log.cc:1196] T 6142bb039cc0454b8b2b691fece15f52 P f6d2ed84d9904640bd42888b0c5779c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/6142bb039cc0454b8b2b691fece15f52
I20240501 20:11:13.039670 21201 catalog_manager.cc:4830] TS f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333): tablet 6142bb039cc0454b8b2b691fece15f52 (table rebalance_test_table_3 [id=67a3850865df4d6f952e084e5e7fc54a]) successfully deleted
I20240501 20:11:13.120450 22067 consensus_queue.cc:237] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:13.121223 22708 raft_consensus.cc:2932] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } } }
I20240501 20:11:13.123370 21491 raft_consensus.cc:1261] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Refusing update from remote peer f5235e7de11c49d8b85ebeb503eeaeae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:13.123960 21199 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d85098d049b4a85ab0f36af6d7a383e with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:11:13.123972 22711 consensus_queue.cc:1035] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b80868c0ff4edabaac58ee1c5b41ac" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 38847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:13.125280 21211 catalog_manager.cc:5484] T 0d85098d049b4a85ab0f36af6d7a383e P 12258cbda618428d9089acc34e1565d3 reported cstate change: config changed from index 4 to 6, VOTER 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "12258cbda618428d9089acc34e1565d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:13.129020 21470 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d85098d049b4a85ab0f36af6d7a383e with delete_type TABLET_DATA_TOMBSTONED (TS 83b80868c0ff4edabaac58ee1c5b41ac not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35790
I20240501 20:11:13.129366 22829 tablet_replica.cc:330] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: stopping tablet replica
I20240501 20:11:13.129555 22829 raft_consensus.cc:2229] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:13.129722 22829 raft_consensus.cc:2258] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:13.130455 22829 ts_tablet_manager.cc:1875] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:13.130803 21356 raft_consensus.cc:1261] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Refusing update from remote peer 12258cbda618428d9089acc34e1565d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:13.131295 22708 consensus_queue.cc:1035] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:13.136097 22829 ts_tablet_manager.cc:1888] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 20:11:13.136198 22829 log.cc:1196] T 0d85098d049b4a85ab0f36af6d7a383e P 83b80868c0ff4edabaac58ee1c5b41ac: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d85098d049b4a85ab0f36af6d7a383e
I20240501 20:11:13.136874 21201 catalog_manager.cc:4830] TS 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847): tablet 0d85098d049b4a85ab0f36af6d7a383e (table rebalance_test_table_2 [id=f0628fbc30d64bd9be053cdd8aaf49f3]) successfully deleted
I20240501 20:11:13.178737 22067 consensus_queue.cc:237] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } }
I20240501 20:11:13.179401 22708 raft_consensus.cc:2932] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } } }
I20240501 20:11:13.181659 21199 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 837412d5b37c4b1791f56e6248283aad with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:11:13.182739 21212 catalog_manager.cc:5484] T 837412d5b37c4b1791f56e6248283aad P 12258cbda618428d9089acc34e1565d3 reported cstate change: config changed from index 3 to 5, VOTER f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "12258cbda618428d9089acc34e1565d3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "12258cbda618428d9089acc34e1565d3" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 34785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:13.185681 21335 tablet_service.cc:1514] Processing DeleteTablet for tablet 837412d5b37c4b1791f56e6248283aad with delete_type TABLET_DATA_TOMBSTONED (TS f6d2ed84d9904640bd42888b0c5779c9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33814
I20240501 20:11:13.185968 22830 tablet_replica.cc:330] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: stopping tablet replica
I20240501 20:11:13.186111 22830 raft_consensus.cc:2229] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:13.186251 22830 raft_consensus.cc:2258] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:13.186941 22830 ts_tablet_manager.cc:1875] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:13.190873 22830 ts_tablet_manager.cc:1888] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 20:11:13.190976 22830 log.cc:1196] T 837412d5b37c4b1791f56e6248283aad P f6d2ed84d9904640bd42888b0c5779c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/837412d5b37c4b1791f56e6248283aad
I20240501 20:11:13.191548 21201 catalog_manager.cc:4830] TS f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333): tablet 837412d5b37c4b1791f56e6248283aad (table TestTable [id=7115296f422a48d2b68808bee3e3d409]) successfully deleted
I20240501 20:11:13.263749 21356 raft_consensus.cc:1261] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Refusing update from remote peer f5235e7de11c49d8b85ebeb503eeaeae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:13.264214 22711 consensus_queue.cc:1035] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d2ed84d9904640bd42888b0c5779c9" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 42333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:13.278036 22199 consensus_queue.cc:237] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:13.278759 22711 raft_consensus.cc:2932] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } } }
I20240501 20:11:13.281056 21201 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6cf57ad231d4706a4a0b62845d90b89 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:11:13.282060 21210 catalog_manager.cc:5484] T f6cf57ad231d4706a4a0b62845d90b89 P f5235e7de11c49d8b85ebeb503eeaeae reported cstate change: config changed from index 3 to 5, VOTER f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:13.284934 21335 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cf57ad231d4706a4a0b62845d90b89 with delete_type TABLET_DATA_TOMBSTONED (TS f6d2ed84d9904640bd42888b0c5779c9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33814
I20240501 20:11:13.285219 22830 tablet_replica.cc:330] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: stopping tablet replica
I20240501 20:11:13.285372 22830 raft_consensus.cc:2229] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:13.285511 22830 raft_consensus.cc:2258] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:13.286151 22830 ts_tablet_manager.cc:1875] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:13.290124 22830 ts_tablet_manager.cc:1888] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 20:11:13.290221 22830 log.cc:1196] T f6cf57ad231d4706a4a0b62845d90b89 P f6d2ed84d9904640bd42888b0c5779c9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/f6cf57ad231d4706a4a0b62845d90b89
I20240501 20:11:13.290817 21201 catalog_manager.cc:4830] TS f6d2ed84d9904640bd42888b0c5779c9 (127.8.68.193:42333): tablet f6cf57ad231d4706a4a0b62845d90b89 (table rebalance_test_table_1 [id=f78e9bb52fb643bb8b49d998c92eeb34]) successfully deleted
I20240501 20:11:13.357456 22199 consensus_queue.cc:237] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } }
I20240501 20:11:13.358315 22475 raft_consensus.cc:2932] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } } }
I20240501 20:11:13.360654 21201 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0309e22baa9e492281e2f35196d9ca5f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:11:13.361877 21210 catalog_manager.cc:5484] T 0309e22baa9e492281e2f35196d9ca5f P f5235e7de11c49d8b85ebeb503eeaeae reported cstate change: config changed from index 3 to 5, VOTER 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5235e7de11c49d8b85ebeb503eeaeae" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:13.364853 21470 tablet_service.cc:1514] Processing DeleteTablet for tablet 0309e22baa9e492281e2f35196d9ca5f with delete_type TABLET_DATA_TOMBSTONED (TS 83b80868c0ff4edabaac58ee1c5b41ac not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35790
I20240501 20:11:13.365139 22829 tablet_replica.cc:330] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: stopping tablet replica
I20240501 20:11:13.365295 22829 raft_consensus.cc:2229] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:13.365448 22829 raft_consensus.cc:2258] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:13.366048 22829 ts_tablet_manager.cc:1875] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:13.370091 22829 ts_tablet_manager.cc:1888] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 20:11:13.370184 22829 log.cc:1196] T 0309e22baa9e492281e2f35196d9ca5f P 83b80868c0ff4edabaac58ee1c5b41ac: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/0309e22baa9e492281e2f35196d9ca5f
I20240501 20:11:13.370807 21201 catalog_manager.cc:4830] TS 83b80868c0ff4edabaac58ee1c5b41ac (127.8.68.194:38847): tablet 0309e22baa9e492281e2f35196d9ca5f (table rebalance_test_table_2 [id=f0628fbc30d64bd9be053cdd8aaf49f3]) successfully deleted
I20240501 20:11:13.573249 22048 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:13.583221 21470 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:13.594339 22179 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:13.604822 21335 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:13.737761 21470 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:13.741191 22048 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:13.749339 22179 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:13.750243 21335 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:14.022733 22048 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:14.023219 21470 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:14.025295 22179 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:14.049355 21335 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 306de9831dd043089fcfd8f0053ea7f6 | 127.8.68.254:43769 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                               Value                                                                                               |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                                              | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                                 | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                              | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                                              | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                              | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                                | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                                               | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.8.68.212:43919 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 12258cbda618428d9089acc34e1565d3 | 127.8.68.195:34785 | HEALTHY | <none>   |       3        |       0
 83b80868c0ff4edabaac58ee1c5b41ac | 127.8.68.194:38847 | HEALTHY | <none>   |       3        |       0
 f5235e7de11c49d8b85ebeb503eeaeae | 127.8.68.196:44331 | HEALTHY | <none>   |       3        |       0
 f6d2ed84d9904640bd42888b0c5779c9 | 127.8.68.193:42333 | HEALTHY | <none>   |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
                   Flag                   |                                                                                             Value                                                                                             |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                                           | experimental          | all 4 server(s) checked
 local_ip_for_outbound_sockets            | 127.8.68.193                                                                                                                                                                                  | experimental          | 127.8.68.193:42333
 local_ip_for_outbound_sockets            | 127.8.68.194                                                                                                                                                                                  | experimental          | 127.8.68.194:38847
 local_ip_for_outbound_sockets            | 127.8.68.195                                                                                                                                                                                  | experimental          | 127.8.68.195:34785
 local_ip_for_outbound_sockets            | 127.8.68.196                                                                                                                                                                                  | experimental          | 127.8.68.196:44331
 never_fsync                              | true                                                                                                                                                                                          | unsafe,advanced       | all 4 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                             | unsafe,hidden         | all 4 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                          | experimental,advanced | all 4 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                          | unsafe                | all 4 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                            | hidden                | all 4 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.8.68.193:42333
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.8.68.194:38847
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.8.68.195:34785
 server_dump_info_path                    | /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.8.68.196:44331

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.8.68.212:43919 | all 4 server(s) checked
 time_source         | builtin            | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 1  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 3
 First Quartile | 3
 Median         | 3
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 5
 Tablets        | 13
 Replicas       | 13

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240501 20:11:14.091313  8467 log_verifier.cc:126] Checking tablet 0309e22baa9e492281e2f35196d9ca5f
I20240501 20:11:14.099119  8467 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 0309e22baa9e492281e2f35196d9ca5f
I20240501 20:11:14.099221  8467 log_verifier.cc:126] Checking tablet 0d85098d049b4a85ab0f36af6d7a383e
I20240501 20:11:14.107182  8467 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0d85098d049b4a85ab0f36af6d7a383e
I20240501 20:11:14.107260  8467 log_verifier.cc:126] Checking tablet 2dff2bedabbf4ae494f535f2e74277cf
I20240501 20:11:14.114094  8467 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2dff2bedabbf4ae494f535f2e74277cf
I20240501 20:11:14.114161  8467 log_verifier.cc:126] Checking tablet 2fcf8e6204d74bfe9604329491dd5ec3
I20240501 20:11:14.121490  8467 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 2fcf8e6204d74bfe9604329491dd5ec3
I20240501 20:11:14.121567  8467 log_verifier.cc:126] Checking tablet 6142bb039cc0454b8b2b691fece15f52
I20240501 20:11:14.129084  8467 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 6142bb039cc0454b8b2b691fece15f52
I20240501 20:11:14.129160  8467 log_verifier.cc:126] Checking tablet 67d8b1dfeecc48d29976027140ba3656
I20240501 20:11:14.136183  8467 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 67d8b1dfeecc48d29976027140ba3656
I20240501 20:11:14.136251  8467 log_verifier.cc:126] Checking tablet 837412d5b37c4b1791f56e6248283aad
I20240501 20:11:14.143821  8467 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 837412d5b37c4b1791f56e6248283aad
I20240501 20:11:14.143889  8467 log_verifier.cc:126] Checking tablet a9e97a6d76434a18a16f554732211897
I20240501 20:11:14.150740  8467 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a9e97a6d76434a18a16f554732211897
I20240501 20:11:14.150804  8467 log_verifier.cc:126] Checking tablet c62a19351f9d49ee95909ad405147c04
I20240501 20:11:14.157924  8467 log_verifier.cc:177] Verified matching terms for 2 ops in tablet c62a19351f9d49ee95909ad405147c04
I20240501 20:11:14.157989  8467 log_verifier.cc:126] Checking tablet e058f2c85c084eff963a95b69dd68117
I20240501 20:11:14.165117  8467 log_verifier.cc:177] Verified matching terms for 2 ops in tablet e058f2c85c084eff963a95b69dd68117
I20240501 20:11:14.165184  8467 log_verifier.cc:126] Checking tablet e5bd1e1b78f0405d9955bbb4c2ac2e44
I20240501 20:11:14.171949  8467 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e5bd1e1b78f0405d9955bbb4c2ac2e44
I20240501 20:11:14.172011  8467 log_verifier.cc:126] Checking tablet f6cf57ad231d4706a4a0b62845d90b89
I20240501 20:11:14.179350  8467 log_verifier.cc:177] Verified matching terms for 5 ops in tablet f6cf57ad231d4706a4a0b62845d90b89
I20240501 20:11:14.179421  8467 log_verifier.cc:126] Checking tablet fad1c1fd383d45dba455731bebfcbbf5
I20240501 20:11:14.187098  8467 log_verifier.cc:177] Verified matching terms for 5 ops in tablet fad1c1fd383d45dba455731bebfcbbf5
I20240501 20:11:14.188165  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 21273
I20240501 20:11:14.189360 21395 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:14.303362  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 21273
W20240501 20:11:14.321444 21198 connection.cc:535] server connection from 127.8.68.193:38067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:14.321565  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 21406
I20240501 20:11:14.322552 21530 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:14.432427  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 21406
I20240501 20:11:14.451004  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 21986
I20240501 20:11:14.456743 22107 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:14.560112  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 21986
I20240501 20:11:14.576319  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 22117
I20240501 20:11:14.582115 22238 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:14.680630  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 22117
I20240501 20:11:14.696501  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 21183
I20240501 20:11:14.697494 21244 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:14.820816  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 21183
2024-05-01T20:11:14Z chronyd exiting
[       OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (16293 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (16293 ms total)

[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN      ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20240501 20:11:14.872035  8467 test_util.cc:268] Using random seed: 380635805
I20240501 20:11:14.873839  8467 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:11:14.873890  8467 ts_itest-base.cc:113] --------------
I20240501 20:11:14.873924  8467 ts_itest-base.cc:114] 6 tablet servers
I20240501 20:11:14.873980  8467 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:11:14.874047  8467 ts_itest-base.cc:116] --------------
2024-05-01T20:11:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:11:14Z Disabled control of system clock
I20240501 20:11:14.925662  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:46623
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:46623
--location_mapping_cmd=/tmp/dist-test-taskH_e3f3/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 20:11:15.240026 22984 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:11:15.240345 22984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:15.240442 22984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:15.240536 22984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:15.247891 22984 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:15.248004 22984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:11:15.248116 22984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:15.248175 22984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:11:15.248255 22984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:11:15.257025 22984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskH_e3f3/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:46623
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:15.258169 22984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:15.259850 22984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:15.269855 22989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:15.270267 22992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:15.270730 22990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:15.271214 22984 server_base.cc:1034] running on GCE node
I20240501 20:11:15.271868 22984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:15.273108 22984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:15.274330 22984 hybrid_clock.cc:648] HybridClock initialized: now 1714594275274251 us; error 38 us; skew 500 ppm
I20240501 20:11:15.274592 22984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:15.276661 22984 webserver.cc:460] Webserver started at http://127.8.68.254:34291/ using document root <none> and password file <none>
I20240501 20:11:15.277269 22984 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:15.277367 22984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:15.277597 22984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:15.279940 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b5f0d27ea2bc42778c1c599a71ab73c5"
format_stamp: "Formatted at 2024-05-01 20:11:15 on dist-test-slave-08p9"
I20240501 20:11:15.280617 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b5f0d27ea2bc42778c1c599a71ab73c5"
format_stamp: "Formatted at 2024-05-01 20:11:15 on dist-test-slave-08p9"
I20240501 20:11:15.285132 22984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240501 20:11:15.288146 22998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:15.289287 22984 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:15.289436 22984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b5f0d27ea2bc42778c1c599a71ab73c5"
format_stamp: "Formatted at 2024-05-01 20:11:15 on dist-test-slave-08p9"
I20240501 20:11:15.289587 22984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:15.305773 22984 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:15.306463 22984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:15.306720 22984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:15.322824 22984 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:46623
I20240501 20:11:15.322863 23049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:46623 every 8 connection(s)
I20240501 20:11:15.323997 22984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:11:15.327164  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 22984
I20240501 20:11:15.327270 23050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:15.327333  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:11:15.340067 23050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5: Bootstrap starting.
I20240501 20:11:15.343142 23050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:15.344120 23050 log.cc:824] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:15.346506 23050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5: No bootstrap required, opened a new log
I20240501 20:11:15.352617 23050 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0d27ea2bc42778c1c599a71ab73c5" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46623 } }
I20240501 20:11:15.352914 23050 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:15.353008 23050 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f0d27ea2bc42778c1c599a71ab73c5, State: Initialized, Role: FOLLOWER
I20240501 20:11:15.353534 23050 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0d27ea2bc42778c1c599a71ab73c5" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46623 } }
I20240501 20:11:15.353725 23050 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:15.353835 23050 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:15.353974 23050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:15.355890 23050 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0d27ea2bc42778c1c599a71ab73c5" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46623 } }
I20240501 20:11:15.356330 23050 leader_election.cc:302] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f0d27ea2bc42778c1c599a71ab73c5; no voters: 
I20240501 20:11:15.356834 23050 leader_election.cc:288] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:15.356943 23055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:15.357483 23055 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [term 1 LEADER]: Becoming Leader. State: Replica: b5f0d27ea2bc42778c1c599a71ab73c5, State: Running, Role: LEADER
I20240501 20:11:15.357996 23055 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0d27ea2bc42778c1c599a71ab73c5" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46623 } }
I20240501 20:11:15.358630 23050 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:11:15.360495 23056 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f0d27ea2bc42778c1c599a71ab73c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0d27ea2bc42778c1c599a71ab73c5" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46623 } } }
I20240501 20:11:15.360831 23056 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:15.360839 23057 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f0d27ea2bc42778c1c599a71ab73c5. Latest consensus state: current_term: 1 leader_uuid: "b5f0d27ea2bc42778c1c599a71ab73c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0d27ea2bc42778c1c599a71ab73c5" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 46623 } } }
I20240501 20:11:15.361676 23057 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5 [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:15.362850 23061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:11:15.368187 23061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:11:15.368813 23073 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240501 20:11:15.375535 23061 catalog_manager.cc:1347] Generated new cluster ID: 91eb02950d294c348c85f952f63a5486
I20240501 20:11:15.375638 23061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:11:15.384821 23061 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:11:15.386335 23061 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:11:15.404074 23061 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b5f0d27ea2bc42778c1c599a71ab73c5: Generated new TSK 0
I20240501 20:11:15.404902 23061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:11:15.411599  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:15.729280 23075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:15.729542 23075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:15.729667 23075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:15.736649 23075 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:15.736765 23075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:15.736912 23075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240501 20:11:15.745267 23075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:15.746400 23075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:15.747978 23075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:15.759042 23080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:15.759399 23081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:15.759913 23083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:15.760421 23075 server_base.cc:1034] running on GCE node
I20240501 20:11:15.760987 23075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:15.761873 23075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:15.763064 23075 hybrid_clock.cc:648] HybridClock initialized: now 1714594275763005 us; error 53 us; skew 500 ppm
I20240501 20:11:15.763415 23075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:15.769863 23075 webserver.cc:460] Webserver started at http://127.8.68.193:39853/ using document root <none> and password file <none>
I20240501 20:11:15.770598 23075 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:15.770714 23075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:15.770998 23075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:15.773445 23075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a4b53d88de2b4e938f65b317b60b0630"
format_stamp: "Formatted at 2024-05-01 20:11:15 on dist-test-slave-08p9"
I20240501 20:11:15.774382 23075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a4b53d88de2b4e938f65b317b60b0630"
format_stamp: "Formatted at 2024-05-01 20:11:15 on dist-test-slave-08p9"
I20240501 20:11:15.778944 23075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240501 20:11:15.781960 23089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:15.783198 23075 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:11:15.783414 23075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a4b53d88de2b4e938f65b317b60b0630"
format_stamp: "Formatted at 2024-05-01 20:11:15 on dist-test-slave-08p9"
I20240501 20:11:15.783618 23075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:15.810900 23075 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:15.811688 23075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:15.811954 23075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:15.812851 23075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:15.814265 23075 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:15.814383 23075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:15.814496 23075 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:15.814576 23075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:15.845111 23075 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:39789
I20240501 20:11:15.845211 23202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:39789 every 8 connection(s)
I20240501 20:11:15.846449 23075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:11:15.852452  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 23075
I20240501 20:11:15.852655  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:11:15.856087  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:15.858791 23203 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46623
I20240501 20:11:15.859082 23203 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:15.859865 23203 heartbeater.cc:511] Master 127.8.68.254:46623 requested a full tablet report, sending...
I20240501 20:11:15.943707 23015 ts_manager.cc:194] Registered new tserver with Master: a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789)
I20240501 20:11:15.946301 23015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:47065
W20240501 20:11:16.184768 23208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:16.185010 23208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:16.185107 23208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:16.192140 23208 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:16.192236 23208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:16.192345 23208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240501 20:11:16.200661 23208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:16.201674 23208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:16.203229 23208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:16.214299 23215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:16.214654 23214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:16.215153 23217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:16.215626 23208 server_base.cc:1034] running on GCE node
I20240501 20:11:16.216118 23208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:16.216908 23208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:16.218091 23208 hybrid_clock.cc:648] HybridClock initialized: now 1714594276218046 us; error 35 us; skew 500 ppm
I20240501 20:11:16.218394 23208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:16.220402 23208 webserver.cc:460] Webserver started at http://127.8.68.194:44461/ using document root <none> and password file <none>
I20240501 20:11:16.221046 23208 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:16.221159 23208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:16.221444 23208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:16.225512 23208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7abd52761123466ca399914a88fb23aa"
format_stamp: "Formatted at 2024-05-01 20:11:16 on dist-test-slave-08p9"
I20240501 20:11:16.226233 23208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7abd52761123466ca399914a88fb23aa"
format_stamp: "Formatted at 2024-05-01 20:11:16 on dist-test-slave-08p9"
I20240501 20:11:16.230769 23208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240501 20:11:16.233695 23223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:16.234819 23208 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.003s
I20240501 20:11:16.234958 23208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7abd52761123466ca399914a88fb23aa"
format_stamp: "Formatted at 2024-05-01 20:11:16 on dist-test-slave-08p9"
I20240501 20:11:16.235126 23208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:16.254061 23208 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:16.254778 23208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:16.255014 23208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:16.255818 23208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:16.257097 23208 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:16.257186 23208 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:16.257289 23208 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:16.257357 23208 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:16.285171 23208 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:41175
I20240501 20:11:16.285226 23337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:41175 every 8 connection(s)
I20240501 20:11:16.286275 23208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:11:16.291002  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 23208
I20240501 20:11:16.291152  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:11:16.294112  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:16.298141 23338 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46623
I20240501 20:11:16.298400 23338 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:16.299106 23338 heartbeater.cc:511] Master 127.8.68.254:46623 requested a full tablet report, sending...
I20240501 20:11:16.353500 23015 ts_manager.cc:194] Registered new tserver with Master: 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:16.354425 23015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:35969
W20240501 20:11:16.624048 23341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:16.624308 23341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:16.624444 23341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:16.631428 23341 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:16.631539 23341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:16.631682 23341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240501 20:11:16.640005 23341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:16.641049 23341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:16.642573 23341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:16.653275 23347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:16.653786 23348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:16.654089 23351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:16.654783 23341 server_base.cc:1034] running on GCE node
I20240501 20:11:16.655287 23341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:16.656100 23341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:16.657298 23341 hybrid_clock.cc:648] HybridClock initialized: now 1714594276657250 us; error 35 us; skew 500 ppm
I20240501 20:11:16.657570 23341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:16.659647 23341 webserver.cc:460] Webserver started at http://127.8.68.195:39489/ using document root <none> and password file <none>
I20240501 20:11:16.660310 23341 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:16.660406 23341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:16.660660 23341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:16.663000 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
format_stamp: "Formatted at 2024-05-01 20:11:16 on dist-test-slave-08p9"
I20240501 20:11:16.663755 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
format_stamp: "Formatted at 2024-05-01 20:11:16 on dist-test-slave-08p9"
I20240501 20:11:16.668378 23341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.004s
I20240501 20:11:16.671317 23357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:16.672492 23341 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:16.672648 23341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
format_stamp: "Formatted at 2024-05-01 20:11:16 on dist-test-slave-08p9"
I20240501 20:11:16.672839 23341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:16.695113 23341 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:16.695808 23341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:16.696040 23341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:16.696866 23341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:16.698176 23341 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:16.698258 23341 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:16.698375 23341 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:16.698446 23341 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:16.726810 23341 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:45603
I20240501 20:11:16.726876 23476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:45603 every 8 connection(s)
I20240501 20:11:16.727931 23341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:11:16.735743  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 23341
I20240501 20:11:16.735913  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:11:16.739046  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:16.739259 23477 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46623
I20240501 20:11:16.739518 23477 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:16.740327 23477 heartbeater.cc:511] Master 127.8.68.254:46623 requested a full tablet report, sending...
I20240501 20:11:16.783198 23015 ts_manager.cc:194] Registered new tserver with Master: ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:16.784451 23015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:53023
I20240501 20:11:16.949483 23203 heartbeater.cc:503] Master 127.8.68.254:46623 was elected leader, sending a full tablet report...
W20240501 20:11:17.058650 23481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:17.058918 23481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:17.059053 23481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:17.066360 23481 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:17.066473 23481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:17.066620 23481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:11:17.075278 23481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:17.076505 23481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:17.078207 23481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:17.089737 23487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:17.090162 23486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:17.090512 23489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:17.091090 23481 server_base.cc:1034] running on GCE node
I20240501 20:11:17.091607 23481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:17.092428 23481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:17.093611 23481 hybrid_clock.cc:648] HybridClock initialized: now 1714594277093563 us; error 41 us; skew 500 ppm
I20240501 20:11:17.093889 23481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:17.095863 23481 webserver.cc:460] Webserver started at http://127.8.68.196:33859/ using document root <none> and password file <none>
I20240501 20:11:17.096496 23481 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:17.096591 23481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:17.096840 23481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:17.099236 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5bd8f7fde1784f5fa1ab24113dc27934"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:17.099905 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5bd8f7fde1784f5fa1ab24113dc27934"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:17.104487 23481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 20:11:17.107434 23495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:17.108562 23481 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:17.108704 23481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5bd8f7fde1784f5fa1ab24113dc27934"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:17.108878 23481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:17.124682 23481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:17.125365 23481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:17.125600 23481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:17.126438 23481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:17.127759 23481 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:17.127851 23481 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:17.127952 23481 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:17.128023 23481 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:17.156409 23481 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:33701
I20240501 20:11:17.156483 23611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:33701 every 8 connection(s)
I20240501 20:11:17.157600 23481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:17.157965  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 23481
I20240501 20:11:17.158138  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:11:17.161763  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:17.169978 23612 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46623
I20240501 20:11:17.170219 23612 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:17.170950 23612 heartbeater.cc:511] Master 127.8.68.254:46623 requested a full tablet report, sending...
I20240501 20:11:17.217515 23015 ts_manager.cc:194] Registered new tserver with Master: 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196:33701)
I20240501 20:11:17.218807 23015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:55301
I20240501 20:11:17.357378 23338 heartbeater.cc:503] Master 127.8.68.254:46623 was elected leader, sending a full tablet report...
W20240501 20:11:17.487057 23615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:17.487301 23615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:17.487409 23615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:17.494544 23615 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:17.494647 23615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:17.494781 23615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:11:17.503194 23615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:17.504277 23615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:17.505863 23615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:17.516981 23621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:17.517463 23622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:17.517793 23624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:17.518425 23615 server_base.cc:1034] running on GCE node
I20240501 20:11:17.518977 23615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:17.519809 23615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:17.520998 23615 hybrid_clock.cc:648] HybridClock initialized: now 1714594277520957 us; error 34 us; skew 500 ppm
I20240501 20:11:17.521286 23615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:17.523383 23615 webserver.cc:460] Webserver started at http://127.8.68.197:45739/ using document root <none> and password file <none>
I20240501 20:11:17.524040 23615 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:17.524158 23615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:17.524454 23615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:17.526875 23615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:17.527603 23615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:17.532284 23615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 20:11:17.535326 23630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:17.536466 23615 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:17.536613 23615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:17.536785 23615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:17.565779 23615 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:17.566602 23615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:17.566851 23615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:17.567714 23615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:17.569121 23615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:17.569208 23615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:17.569315 23615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:17.569386 23615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:17.597153 23615 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:33911
I20240501 20:11:17.597225 23744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:33911 every 8 connection(s)
I20240501 20:11:17.598380 23615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:11:17.602775  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 23615
I20240501 20:11:17.602999  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:11:17.606530  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:0
--local_ip_for_outbound_sockets=127.8.68.198
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:17.611301 23745 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46623
I20240501 20:11:17.611593 23745 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:17.612505 23745 heartbeater.cc:511] Master 127.8.68.254:46623 requested a full tablet report, sending...
I20240501 20:11:17.655483 23015 ts_manager.cc:194] Registered new tserver with Master: 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197:33911)
I20240501 20:11:17.656361 23015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:53325
I20240501 20:11:17.787464 23477 heartbeater.cc:503] Master 127.8.68.254:46623 was elected leader, sending a full tablet report...
W20240501 20:11:17.918561 23748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:17.918821 23748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:17.918958 23748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:17.925911 23748 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:17.926028 23748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:17.926174 23748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240501 20:11:17.935305 23748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=0
--tserver_master_addrs=127.8.68.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:17.936414 23748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:17.937939 23748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:17.949059 23754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:17.949573 23757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:17.950031 23755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:17.950572 23748 server_base.cc:1034] running on GCE node
I20240501 20:11:17.951108 23748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:17.952030 23748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:17.953248 23748 hybrid_clock.cc:648] HybridClock initialized: now 1714594277953195 us; error 53 us; skew 500 ppm
I20240501 20:11:17.953590 23748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:17.955693 23748 webserver.cc:460] Webserver started at http://127.8.68.198:43189/ using document root <none> and password file <none>
I20240501 20:11:17.956332 23748 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:17.956431 23748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:17.956688 23748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:17.959102 23748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "527d2c1d7b3d45aeb992646b89911c14"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:17.959784 23748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "527d2c1d7b3d45aeb992646b89911c14"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:17.964493 23748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 20:11:17.967512 23763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:17.968693 23748 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:17.968833 23748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "527d2c1d7b3d45aeb992646b89911c14"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:17.969007 23748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:17.984723 23748 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:17.985424 23748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:17.985663 23748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:17.986495 23748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:17.987802 23748 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:17.987892 23748 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:17.987990 23748 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:17.988070 23748 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:18.018318 23748 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:43689
I20240501 20:11:18.018397 23878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:43689 every 8 connection(s)
I20240501 20:11:18.019516 23748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:11:18.025683  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 23748
I20240501 20:11:18.025836  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:11:18.031628 23879 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46623
I20240501 20:11:18.031888 23879 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:18.032631 23879 heartbeater.cc:511] Master 127.8.68.254:46623 requested a full tablet report, sending...
I20240501 20:11:18.073668 23015 ts_manager.cc:194] Registered new tserver with Master: 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689)
I20240501 20:11:18.074600 23015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:52431
I20240501 20:11:18.078613  8467 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 20:11:18.094368  8467 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 20:11:18.094529  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 23481
I20240501 20:11:18.102273 23607 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:18.180060  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 23481
I20240501 20:11:18.193671  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 23748
I20240501 20:11:18.201501 23874 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:18.279168  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 23748
I20240501 20:11:18.292501  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 23615
I20240501 20:11:18.300000 23740 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:18.378175  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 23615
I20240501 20:11:22.143579 23015 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58642:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 20:11:22.150038 23015 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:11:22.184917 23407 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa (DEFAULT_TABLE table=rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:22.185573 23409 tablet_service.cc:1467] Processing CreateTablet for tablet b7a6b264dacb4038b840d9b1d033641d (DEFAULT_TABLE table=rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:22.185317 23410 tablet_service.cc:1467] Processing CreateTablet for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:22.186713 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.187215 23410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.189397 23409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a6b264dacb4038b840d9b1d033641d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.190459 23272 tablet_service.cc:1467] Processing CreateTablet for tablet b7a6b264dacb4038b840d9b1d033641d (DEFAULT_TABLE table=rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:22.191187 23137 tablet_service.cc:1467] Processing CreateTablet for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:22.191253 23273 tablet_service.cc:1467] Processing CreateTablet for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:22.191546 23271 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa (DEFAULT_TABLE table=rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:22.192301 23137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.192481 23272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a6b264dacb4038b840d9b1d033641d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.190922 23136 tablet_service.cc:1467] Processing CreateTablet for tablet b7a6b264dacb4038b840d9b1d033641d (DEFAULT_TABLE table=rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:22.193930 23136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a6b264dacb4038b840d9b1d033641d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.194586 23273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.195199 23271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.191025 23135 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa (DEFAULT_TABLE table=rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:22.196048 23135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.210515 23909 tablet_bootstrap.cc:492] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:22.215339 23907 tablet_bootstrap.cc:492] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9: Bootstrap starting.
I20240501 20:11:22.216320 23908 tablet_bootstrap.cc:492] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:22.218497 23907 tablet_bootstrap.cc:654] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:22.214273 23909 tablet_bootstrap.cc:654] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:22.219439 23908 tablet_bootstrap.cc:654] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:22.219610 23909 log.cc:824] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:22.219707 23907 log.cc:824] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:22.220549 23908 log.cc:824] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:22.224819 23908 tablet_bootstrap.cc:492] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630: No bootstrap required, opened a new log
I20240501 20:11:22.225109 23908 ts_tablet_manager.cc:1367] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.002s
I20240501 20:11:22.226943 23907 tablet_bootstrap.cc:492] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9: No bootstrap required, opened a new log
I20240501 20:11:22.227170 23907 ts_tablet_manager.cc:1367] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.010s
I20240501 20:11:22.227197 23909 tablet_bootstrap.cc:492] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: No bootstrap required, opened a new log
I20240501 20:11:22.227427 23909 ts_tablet_manager.cc:1367] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.000s
I20240501 20:11:22.232540 23907 raft_consensus.cc:348] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.232846 23907 raft_consensus.cc:374] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:22.232944 23907 raft_consensus.cc:729] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Initialized, Role: FOLLOWER
I20240501 20:11:22.233489 23907 consensus_queue.cc:260] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.233505 23908 raft_consensus.cc:348] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.233878 23908 raft_consensus.cc:374] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:22.233976 23908 raft_consensus.cc:729] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: FOLLOWER
I20240501 20:11:22.234989 23907 ts_tablet_manager.cc:1398] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20240501 20:11:22.235328 23908 consensus_queue.cc:260] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.235498 23907 tablet_bootstrap.cc:492] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9: Bootstrap starting.
I20240501 20:11:22.235567 23909 raft_consensus.cc:348] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.235947 23909 raft_consensus.cc:374] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:22.236056 23909 raft_consensus.cc:729] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: FOLLOWER
I20240501 20:11:22.236658 23909 consensus_queue.cc:260] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.237327 23907 tablet_bootstrap.cc:654] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:22.243069 23908 ts_tablet_manager.cc:1398] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.018s	user 0.010s	sys 0.005s
I20240501 20:11:22.243321 23907 tablet_bootstrap.cc:492] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9: No bootstrap required, opened a new log
I20240501 20:11:22.243453 23907 ts_tablet_manager.cc:1367] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240501 20:11:22.243587 23908 tablet_bootstrap.cc:492] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:22.244077 23909 ts_tablet_manager.cc:1398] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.008s
I20240501 20:11:22.244429 23909 tablet_bootstrap.cc:492] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:22.244333 23907 raft_consensus.cc:348] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.244510 23907 raft_consensus.cc:374] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:22.244609 23907 raft_consensus.cc:729] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Initialized, Role: FOLLOWER
I20240501 20:11:22.244820 23907 consensus_queue.cc:260] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.245277 23907 ts_tablet_manager.cc:1398] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:11:22.245468 23908 tablet_bootstrap.cc:654] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:22.245556 23907 tablet_bootstrap.cc:492] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9: Bootstrap starting.
I20240501 20:11:22.245985 23909 tablet_bootstrap.cc:654] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:22.247010 23907 tablet_bootstrap.cc:654] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:22.255280 23909 tablet_bootstrap.cc:492] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa: No bootstrap required, opened a new log
I20240501 20:11:22.255412 23909 ts_tablet_manager.cc:1367] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.006s
I20240501 20:11:22.255452 23908 tablet_bootstrap.cc:492] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630: No bootstrap required, opened a new log
I20240501 20:11:22.255592 23908 ts_tablet_manager.cc:1367] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240501 20:11:22.256178 23909 raft_consensus.cc:348] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } }
I20240501 20:11:22.256331 23909 raft_consensus.cc:374] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:22.256421 23909 raft_consensus.cc:729] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: FOLLOWER
I20240501 20:11:22.256554 23908 raft_consensus.cc:348] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.256659 23909 consensus_queue.cc:260] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } }
I20240501 20:11:22.256762 23908 raft_consensus.cc:374] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:22.256842 23908 raft_consensus.cc:729] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: FOLLOWER
I20240501 20:11:22.257135 23909 ts_tablet_manager.cc:1398] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:11:22.257360 23909 tablet_bootstrap.cc:492] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:22.257840 23907 tablet_bootstrap.cc:492] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9: No bootstrap required, opened a new log
I20240501 20:11:22.257999 23907 ts_tablet_manager.cc:1367] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.008s
I20240501 20:11:22.258669 23909 tablet_bootstrap.cc:654] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:22.258806 23907 raft_consensus.cc:348] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } }
I20240501 20:11:22.259007 23907 raft_consensus.cc:374] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:22.259359 23907 raft_consensus.cc:729] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Initialized, Role: FOLLOWER
I20240501 20:11:22.259574 23907 consensus_queue.cc:260] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } }
I20240501 20:11:22.259953 23908 consensus_queue.cc:260] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.260433 23908 ts_tablet_manager.cc:1398] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240501 20:11:22.260702 23908 tablet_bootstrap.cc:492] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:22.261924 23908 tablet_bootstrap.cc:654] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:22.262656 23907 ts_tablet_manager.cc:1398] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 20:11:22.266669 23909 tablet_bootstrap.cc:492] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa: No bootstrap required, opened a new log
I20240501 20:11:22.266824 23909 ts_tablet_manager.cc:1367] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.009s	user 0.008s	sys 0.001s
I20240501 20:11:22.267279 23908 tablet_bootstrap.cc:492] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630: No bootstrap required, opened a new log
I20240501 20:11:22.267314 23909 raft_consensus.cc:348] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.267412 23908 ts_tablet_manager.cc:1367] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240501 20:11:22.267472 23909 raft_consensus.cc:374] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:22.267522 23909 raft_consensus.cc:729] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: FOLLOWER
I20240501 20:11:22.267707 23909 consensus_queue.cc:260] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.267944 23908 raft_consensus.cc:348] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } }
I20240501 20:11:22.268141 23909 ts_tablet_manager.cc:1398] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:22.268157 23908 raft_consensus.cc:374] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:22.268229 23908 raft_consensus.cc:729] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: FOLLOWER
I20240501 20:11:22.268446 23908 consensus_queue.cc:260] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } }
I20240501 20:11:22.269004 23908 ts_tablet_manager.cc:1398] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
W20240501 20:11:22.291371 23339 tablet.cc:2369] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:11:22.352059 23204 tablet.cc:2369] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:11:22.355943 23913 raft_consensus.cc:482] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:22.356153 23913 raft_consensus.cc:504] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.357733 23913 leader_election.cc:288] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:22.364450 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a6b264dacb4038b840d9b1d033641d" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630" is_pre_election: true
I20240501 20:11:22.364461 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a6b264dacb4038b840d9b1d033641d" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abd52761123466ca399914a88fb23aa" is_pre_election: true
I20240501 20:11:22.364871 23157 raft_consensus.cc:2454] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 0.
I20240501 20:11:22.364882 23293 raft_consensus.cc:2454] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 0.
I20240501 20:11:22.365401 23361 leader_election.cc:302] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:22.365726 23913 raft_consensus.cc:2781] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:22.365871 23913 raft_consensus.cc:482] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:22.365967 23913 raft_consensus.cc:3037] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:22.367945 23913 raft_consensus.cc:504] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.368506 23913 leader_election.cc:288] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 election: Requested vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:22.368906 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a6b264dacb4038b840d9b1d033641d" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:22.369020 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a6b264dacb4038b840d9b1d033641d" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abd52761123466ca399914a88fb23aa"
I20240501 20:11:22.369096 23157 raft_consensus.cc:3037] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:22.369213 23293 raft_consensus.cc:3037] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:22.372021 23157 raft_consensus.cc:2454] T b7a6b264dacb4038b840d9b1d033641d P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 1.
I20240501 20:11:22.372033 23293 raft_consensus.cc:2454] T b7a6b264dacb4038b840d9b1d033641d P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 1.
I20240501 20:11:22.372400 23361 leader_election.cc:302] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:22.372774 23913 raft_consensus.cc:2781] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:22.373181 23913 raft_consensus.cc:686] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Becoming Leader. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Running, Role: LEADER
I20240501 20:11:22.373648 23913 consensus_queue.cc:237] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.379218 23015 catalog_manager.cc:5484] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:22.413897 23914 raft_consensus.cc:482] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:22.414042 23914 raft_consensus.cc:504] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.415480 23914 leader_election.cc:288] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:22.421674 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0b80ffdd2d4d1c8f6d6c106c724efa" candidate_uuid: "a4b53d88de2b4e938f65b317b60b0630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abd52761123466ca399914a88fb23aa" is_pre_election: true
I20240501 20:11:22.421867 23293 raft_consensus.cc:2454] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b53d88de2b4e938f65b317b60b0630 in term 0.
I20240501 20:11:22.421860 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0b80ffdd2d4d1c8f6d6c106c724efa" candidate_uuid: "a4b53d88de2b4e938f65b317b60b0630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" is_pre_election: true
I20240501 20:11:22.422214 23432 raft_consensus.cc:2454] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b53d88de2b4e938f65b317b60b0630 in term 0.
I20240501 20:11:22.422261 23093 leader_election.cc:302] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, a4b53d88de2b4e938f65b317b60b0630; no voters: 
I20240501 20:11:22.422677 23914 raft_consensus.cc:2781] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:22.422782 23914 raft_consensus.cc:482] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:22.422876 23914 raft_consensus.cc:3037] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:22.424686 23914 raft_consensus.cc:504] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.425181 23914 leader_election.cc:288] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [CANDIDATE]: Term 1 election: Requested vote from peers 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:22.425549 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0b80ffdd2d4d1c8f6d6c106c724efa" candidate_uuid: "a4b53d88de2b4e938f65b317b60b0630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abd52761123466ca399914a88fb23aa"
I20240501 20:11:22.425611 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0b80ffdd2d4d1c8f6d6c106c724efa" candidate_uuid: "a4b53d88de2b4e938f65b317b60b0630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
I20240501 20:11:22.425726 23293 raft_consensus.cc:3037] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:22.425757 23432 raft_consensus.cc:3037] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:22.427512 23293 raft_consensus.cc:2454] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b53d88de2b4e938f65b317b60b0630 in term 1.
I20240501 20:11:22.427610 23432 raft_consensus.cc:2454] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b53d88de2b4e938f65b317b60b0630 in term 1.
I20240501 20:11:22.427858 23915 raft_consensus.cc:482] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:22.427935 23093 leader_election.cc:302] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, a4b53d88de2b4e938f65b317b60b0630; no voters: 
I20240501 20:11:22.428011 23915 raft_consensus.cc:504] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } }
I20240501 20:11:22.428212 23914 raft_consensus.cc:2781] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:22.428617 23914 raft_consensus.cc:686] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEADER]: Becoming Leader. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Running, Role: LEADER
I20240501 20:11:22.429028 23914 consensus_queue.cc:237] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:22.429495 23915 leader_election.cc:288] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603), a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789)
I20240501 20:11:22.433997 23015 catalog_manager.cc:5484] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "a4b53d88de2b4e938f65b317b60b0630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:22.436520 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9d141aa6ef4fa190ae8b9e7f4a4ec8" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" is_pre_election: true
I20240501 20:11:22.436725 23432 raft_consensus.cc:2454] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:22.437176 23224 leader_election.cc:302] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:22.437513 23915 raft_consensus.cc:2781] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:22.437642 23915 raft_consensus.cc:482] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:22.437747 23915 raft_consensus.cc:3037] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:22.437758 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9d141aa6ef4fa190ae8b9e7f4a4ec8" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630" is_pre_election: true
I20240501 20:11:22.437965 23157 raft_consensus.cc:2454] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:22.440034 23915 raft_consensus.cc:504] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } }
I20240501 20:11:22.440574 23915 leader_election.cc:288] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Requested vote from peers ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603), a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789)
I20240501 20:11:22.440894 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9d141aa6ef4fa190ae8b9e7f4a4ec8" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:22.440896 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9d141aa6ef4fa190ae8b9e7f4a4ec8" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
I20240501 20:11:22.441052 23157 raft_consensus.cc:3037] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:22.441092 23432 raft_consensus.cc:3037] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:22.442909 23157 raft_consensus.cc:2454] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:22.442911 23432 raft_consensus.cc:2454] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:22.443269 23224 leader_election.cc:302] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:22.443534 23915 raft_consensus.cc:2781] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:22.443912 23915 raft_consensus.cc:686] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Becoming Leader. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Running, Role: LEADER
I20240501 20:11:22.444330 23915 consensus_queue.cc:237] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } }
I20240501 20:11:22.448154 23015 catalog_manager.cc:5484] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 7abd52761123466ca399914a88fb23aa (127.8.68.194). New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } }
W20240501 20:11:22.482925 23478 tablet.cc:2369] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:11:22.765180 23934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:22.765458 23934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:22.772785 23934 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 20:11:22.809768 23913 consensus_queue.cc:1035] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:22.820565 23913 consensus_queue.cc:1035] T b7a6b264dacb4038b840d9b1d033641d P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:22.855825 23915 consensus_queue.cc:1035] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:22.871790 23932 consensus_queue.cc:1035] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:22.887753 23914 consensus_queue.cc:1035] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:22.901715 23914 consensus_queue.cc:1035] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 235.046 ms
  time per row: 0.117523 ms
I20240501 20:11:23.139847 23013 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58642:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 20:11:23.140552 23013 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:11:23.148308 23272 tablet_service.cc:1467] Processing CreateTablet for tablet 6c56eb0a04104909ae46e1c7c8d732c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:23.148830 23272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c56eb0a04104909ae46e1c7c8d732c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.148721 23409 tablet_service.cc:1467] Processing CreateTablet for tablet 6c56eb0a04104909ae46e1c7c8d732c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:23.149132 23409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c56eb0a04104909ae46e1c7c8d732c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.149354 23136 tablet_service.cc:1467] Processing CreateTablet for tablet 6c56eb0a04104909ae46e1c7c8d732c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:23.149773 23136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c56eb0a04104909ae46e1c7c8d732c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.149889 23271 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb01b1a8a02483e8d49cf2327886f18 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:23.150271 23271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb01b1a8a02483e8d49cf2327886f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.151150 23137 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb01b1a8a02483e8d49cf2327886f18 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:23.151490 23137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb01b1a8a02483e8d49cf2327886f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.154672 23135 tablet_service.cc:1467] Processing CreateTablet for tablet 7475f202d7f2476fbd69ef838d7e1ee2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:23.154997 23135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7475f202d7f2476fbd69ef838d7e1ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.155401 23410 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb01b1a8a02483e8d49cf2327886f18 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:23.155748 23410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb01b1a8a02483e8d49cf2327886f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.157040 23407 tablet_service.cc:1467] Processing CreateTablet for tablet 7475f202d7f2476fbd69ef838d7e1ee2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:23.157405 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7475f202d7f2476fbd69ef838d7e1ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.163519 23994 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:23.165457 23994 tablet_bootstrap.cc:654] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:23.165551 23995 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:23.167541 23995 tablet_bootstrap.cc:654] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:23.167986 23994 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa: No bootstrap required, opened a new log
I20240501 20:11:23.168203 23994 ts_tablet_manager.cc:1367] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240501 20:11:23.168462 23271 tablet_service.cc:1467] Processing CreateTablet for tablet 7475f202d7f2476fbd69ef838d7e1ee2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:23.168891 23271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7475f202d7f2476fbd69ef838d7e1ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.169325 23994 raft_consensus.cc:348] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.169569 23994 raft_consensus.cc:374] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:23.169685 23994 raft_consensus.cc:729] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: FOLLOWER
I20240501 20:11:23.169970 23994 consensus_queue.cc:260] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.170683 23994 ts_tablet_manager.cc:1398] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:11:23.171008 23994 tablet_bootstrap.cc:492] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:23.172667 23994 tablet_bootstrap.cc:654] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:23.178591 23994 tablet_bootstrap.cc:492] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa: No bootstrap required, opened a new log
I20240501 20:11:23.178825 23994 ts_tablet_manager.cc:1367] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.002s
I20240501 20:11:23.179695 23994 raft_consensus.cc:348] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.180008 23994 raft_consensus.cc:374] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:23.180127 23994 raft_consensus.cc:729] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: FOLLOWER
I20240501 20:11:23.180461 23994 consensus_queue.cc:260] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.181752 23993 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: Bootstrap starting.
I20240501 20:11:23.182560 23994 ts_tablet_manager.cc:1398] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:11:23.183431 23995 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: No bootstrap required, opened a new log
I20240501 20:11:23.183470 23994 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:23.183665 23995 ts_tablet_manager.cc:1367] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.018s	user 0.007s	sys 0.000s
I20240501 20:11:23.183722 23993 tablet_bootstrap.cc:654] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:23.184571 23995 raft_consensus.cc:348] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.184864 23995 raft_consensus.cc:374] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:23.184921 23994 tablet_bootstrap.cc:654] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:23.185159 23995 raft_consensus.cc:729] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: FOLLOWER
I20240501 20:11:23.185472 23995 consensus_queue.cc:260] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.186295 23995 ts_tablet_manager.cc:1398] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:23.186609 23995 tablet_bootstrap.cc:492] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:23.188587 23995 tablet_bootstrap.cc:654] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:23.192138 23995 tablet_bootstrap.cc:492] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630: No bootstrap required, opened a new log
I20240501 20:11:23.192282 23995 ts_tablet_manager.cc:1367] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.002s
I20240501 20:11:23.192487 23994 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: No bootstrap required, opened a new log
I20240501 20:11:23.192618 23994 ts_tablet_manager.cc:1367] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240501 20:11:23.193123 23993 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: No bootstrap required, opened a new log
I20240501 20:11:23.193061 23995 raft_consensus.cc:348] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.193274 23993 ts_tablet_manager.cc:1367] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240501 20:11:23.193239 23994 raft_consensus.cc:348] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.193364 23995 raft_consensus.cc:374] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:23.193434 23994 raft_consensus.cc:374] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:23.193496 23995 raft_consensus.cc:729] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: FOLLOWER
I20240501 20:11:23.193526 23994 raft_consensus.cc:729] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: FOLLOWER
I20240501 20:11:23.193714 23994 consensus_queue.cc:260] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.193775 23995 consensus_queue.cc:260] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.194088 23993 raft_consensus.cc:348] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.194319 23993 raft_consensus.cc:374] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:23.194409 23995 ts_tablet_manager.cc:1398] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:11:23.194420 23993 raft_consensus.cc:729] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Initialized, Role: FOLLOWER
I20240501 20:11:23.194778 23995 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:23.194658 23993 consensus_queue.cc:260] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.195211 23993 ts_tablet_manager.cc:1398] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:11:23.195511 23993 tablet_bootstrap.cc:492] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9: Bootstrap starting.
I20240501 20:11:23.197117 23993 tablet_bootstrap.cc:654] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:23.198786 23994 ts_tablet_manager.cc:1398] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 20:11:23.202698 23995 tablet_bootstrap.cc:654] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:23.205431 23995 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: No bootstrap required, opened a new log
I20240501 20:11:23.205525 23993 tablet_bootstrap.cc:492] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9: No bootstrap required, opened a new log
I20240501 20:11:23.205551 23995 ts_tablet_manager.cc:1367] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240501 20:11:23.205691 23993 ts_tablet_manager.cc:1367] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent bootstrapping tablet: real 0.010s	user 0.001s	sys 0.009s
I20240501 20:11:23.206075 23995 raft_consensus.cc:348] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.206215 23995 raft_consensus.cc:374] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:23.206318 23995 raft_consensus.cc:729] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: FOLLOWER
I20240501 20:11:23.206307 23993 raft_consensus.cc:348] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.206516 23993 raft_consensus.cc:374] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:23.206518 23995 consensus_queue.cc:260] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.206661 23993 raft_consensus.cc:729] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Initialized, Role: FOLLOWER
I20240501 20:11:23.206887 23993 consensus_queue.cc:260] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.207177 23995 ts_tablet_manager.cc:1398] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:11:23.207401 23993 ts_tablet_manager.cc:1398] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:11:23.207645 23993 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9: Bootstrap starting.
I20240501 20:11:23.209481 23993 tablet_bootstrap.cc:654] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:23.211524 23993 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9: No bootstrap required, opened a new log
I20240501 20:11:23.211686 23993 ts_tablet_manager.cc:1367] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 20:11:23.212267 23993 raft_consensus.cc:348] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.212414 23993 raft_consensus.cc:374] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:23.212512 23993 raft_consensus.cc:729] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Initialized, Role: FOLLOWER
I20240501 20:11:23.212718 23993 consensus_queue.cc:260] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.213131 23993 ts_tablet_manager.cc:1398] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:23.269117 23924 raft_consensus.cc:482] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:23.269253 23924 raft_consensus.cc:504] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.269852 23924 leader_election.cc:288] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:23.270293 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb01b1a8a02483e8d49cf2327886f18" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630" is_pre_election: true
I20240501 20:11:23.270359 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb01b1a8a02483e8d49cf2327886f18" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abd52761123466ca399914a88fb23aa" is_pre_election: true
I20240501 20:11:23.270506 23157 raft_consensus.cc:2454] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 0.
I20240501 20:11:23.270550 23293 raft_consensus.cc:2454] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 0.
I20240501 20:11:23.270884 23360 leader_election.cc:302] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b53d88de2b4e938f65b317b60b0630, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:23.271129 23924 raft_consensus.cc:2781] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:23.271226 23924 raft_consensus.cc:482] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:23.271288 23924 raft_consensus.cc:3037] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:23.273034 23924 raft_consensus.cc:504] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.273677 23924 leader_election.cc:288] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 election: Requested vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:23.273984 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb01b1a8a02483e8d49cf2327886f18" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:23.274147 23157 raft_consensus.cc:3037] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:23.274345 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb01b1a8a02483e8d49cf2327886f18" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abd52761123466ca399914a88fb23aa"
I20240501 20:11:23.274518 23293 raft_consensus.cc:3037] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:23.276036 23157 raft_consensus.cc:2454] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 1.
I20240501 20:11:23.276394 23360 leader_election.cc:302] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b53d88de2b4e938f65b317b60b0630, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:23.276712 23293 raft_consensus.cc:2454] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 1.
I20240501 20:11:23.276724 23924 raft_consensus.cc:2781] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:23.276886 23924 raft_consensus.cc:686] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Becoming Leader. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Running, Role: LEADER
I20240501 20:11:23.277211 23924 consensus_queue.cc:237] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.279647 23014 catalog_manager.cc:5484] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:23.293010 23987 raft_consensus.cc:482] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:23.293154 23987 raft_consensus.cc:504] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.293929 23987 leader_election.cc:288] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:23.294343 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7475f202d7f2476fbd69ef838d7e1ee2" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630" is_pre_election: true
I20240501 20:11:23.294428 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7475f202d7f2476fbd69ef838d7e1ee2" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" is_pre_election: true
I20240501 20:11:23.294602 23432 raft_consensus.cc:2454] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:23.294647 23157 raft_consensus.cc:2454] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:23.294981 23224 leader_election.cc:302] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:23.295221 23987 raft_consensus.cc:2781] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:23.295333 23987 raft_consensus.cc:482] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:23.295428 23987 raft_consensus.cc:3037] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:23.297307 23987 raft_consensus.cc:504] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.297806 23987 leader_election.cc:288] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Requested vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:23.298139 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7475f202d7f2476fbd69ef838d7e1ee2" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:23.298352 23157 raft_consensus.cc:3037] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:23.298353 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7475f202d7f2476fbd69ef838d7e1ee2" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
I20240501 20:11:23.298528 23432 raft_consensus.cc:3037] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:23.301007 23157 raft_consensus.cc:2454] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:23.301035 23432 raft_consensus.cc:2454] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:23.301406 23226 leader_election.cc:302] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, a4b53d88de2b4e938f65b317b60b0630; no voters: 
I20240501 20:11:23.301702 23987 raft_consensus.cc:2781] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:23.301849 23987 raft_consensus.cc:686] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Becoming Leader. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Running, Role: LEADER
I20240501 20:11:23.302074 23987 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.304572 23014 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 7abd52761123466ca399914a88fb23aa (127.8.68.194). New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:23.381978 23924 raft_consensus.cc:482] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:23.382181 23924 raft_consensus.cc:504] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.382844 23924 leader_election.cc:288] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175), a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789)
I20240501 20:11:23.383349 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c56eb0a04104909ae46e1c7c8d732c0" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630" is_pre_election: true
I20240501 20:11:23.383379 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c56eb0a04104909ae46e1c7c8d732c0" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abd52761123466ca399914a88fb23aa" is_pre_election: true
I20240501 20:11:23.383559 23157 raft_consensus.cc:2454] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 0.
I20240501 20:11:23.383605 23293 raft_consensus.cc:2454] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 0.
I20240501 20:11:23.383996 23360 leader_election.cc:302] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b53d88de2b4e938f65b317b60b0630, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:23.384275 23924 raft_consensus.cc:2781] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:23.384359 23924 raft_consensus.cc:482] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:23.384455 23924 raft_consensus.cc:3037] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:23.386243 23924 raft_consensus.cc:504] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.386829 23924 leader_election.cc:288] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175), a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789)
I20240501 20:11:23.387199 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c56eb0a04104909ae46e1c7c8d732c0" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abd52761123466ca399914a88fb23aa"
I20240501 20:11:23.387288 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c56eb0a04104909ae46e1c7c8d732c0" candidate_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:23.387410 23293 raft_consensus.cc:3037] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:23.387477 23157 raft_consensus.cc:3037] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:23.390074 23293 raft_consensus.cc:2454] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 1.
I20240501 20:11:23.390115 23157 raft_consensus.cc:2454] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb07a45c9604747986a01ae6f7f7bc9 in term 1.
I20240501 20:11:23.390455 23360 leader_election.cc:302] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b53d88de2b4e938f65b317b60b0630, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:23.390705 23924 raft_consensus.cc:2781] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:23.390854 23924 raft_consensus.cc:686] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Becoming Leader. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Running, Role: LEADER
I20240501 20:11:23.391099 23924 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:23.393549 23014 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } }
W20240501 20:11:23.713833 24008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:23.714097 24008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:23.721149 24008 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:11:23.783067 23924 consensus_queue.cc:1035] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:23.784574 23913 consensus_queue.cc:1035] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:23.829752 23913 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:23.840898 23293 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:23.841607 23913 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:23.903016 23432 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:23.903683 23987 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:23.907220 23157 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:23.907742 23987 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:23.922634 24034 mvcc.cc:205] Tried to move back new op lower bound from 7022978186771542016 to 7022978184407597056. Current Snapshot: MvccSnapshot[applied={T|T < 7022978186771542016}]
I20240501 20:11:23.924374 24033 mvcc.cc:205] Tried to move back new op lower bound from 7022978186771542016 to 7022978184407597056. Current Snapshot: MvccSnapshot[applied={T|T < 7022978186771542016}]

INSERT report
    rows total: 2000
    time total: 202.337 ms
  time per row: 0.101169 ms
I20240501 20:11:23.986644 24053 mvcc.cc:205] Tried to move back new op lower bound from 7022978186771542016 to 7022978184407597056. Current Snapshot: MvccSnapshot[applied={T|T < 7022978186771542016}]
I20240501 20:11:24.075740 23014 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58642:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 20:11:24.076463 23014 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:11:24.084313 23271 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3d2792048e4b4aa59c034e0539ddb7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:24.084376 23136 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3d2792048e4b4aa59c034e0539ddb7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:24.084520 23409 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3d2792048e4b4aa59c034e0539ddb7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:24.084758 23271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3d2792048e4b4aa59c034e0539ddb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:24.084800 23136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3d2792048e4b4aa59c034e0539ddb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:24.084901 23409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3d2792048e4b4aa59c034e0539ddb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:24.086717 23407 tablet_service.cc:1467] Processing CreateTablet for tablet 6155d934dae348c99ffb489cefcd64b4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:24.087057 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155d934dae348c99ffb489cefcd64b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:24.090700 23410 tablet_service.cc:1467] Processing CreateTablet for tablet d940d4d4f1a54388b49f6794ca57d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:24.091050 23410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940d4d4f1a54388b49f6794ca57d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:24.095166 23272 tablet_service.cc:1467] Processing CreateTablet for tablet 6155d934dae348c99ffb489cefcd64b4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:24.095188 23273 tablet_service.cc:1467] Processing CreateTablet for tablet d940d4d4f1a54388b49f6794ca57d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:24.095561 23273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940d4d4f1a54388b49f6794ca57d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:24.096117 23137 tablet_service.cc:1467] Processing CreateTablet for tablet 6155d934dae348c99ffb489cefcd64b4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:24.096503 23137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155d934dae348c99ffb489cefcd64b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:24.100615 24070 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:24.101419 23272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155d934dae348c99ffb489cefcd64b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:24.101994 23135 tablet_service.cc:1467] Processing CreateTablet for tablet d940d4d4f1a54388b49f6794ca57d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:24.102336 24070 tablet_bootstrap.cc:654] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:24.102351 23135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940d4d4f1a54388b49f6794ca57d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:24.102435 24069 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9: Bootstrap starting.
I20240501 20:11:24.104396 24069 tablet_bootstrap.cc:654] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:24.105581 24068 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:24.107631 24068 tablet_bootstrap.cc:654] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:24.108685 24069 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9: No bootstrap required, opened a new log
I20240501 20:11:24.108879 24069 ts_tablet_manager.cc:1367] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.003s
I20240501 20:11:24.108922 24070 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: No bootstrap required, opened a new log
I20240501 20:11:24.109038 24070 ts_tablet_manager.cc:1367] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.003s
I20240501 20:11:24.109750 24069 raft_consensus.cc:348] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.109753 24070 raft_consensus.cc:348] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.109941 24070 raft_consensus.cc:374] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:24.110023 24070 raft_consensus.cc:729] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: FOLLOWER
I20240501 20:11:24.110057 24069 raft_consensus.cc:374] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:24.110249 24069 raft_consensus.cc:729] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Initialized, Role: FOLLOWER
I20240501 20:11:24.110234 24070 consensus_queue.cc:260] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.110829 24070 ts_tablet_manager.cc:1398] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:24.111060 24070 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:24.112428 24070 tablet_bootstrap.cc:654] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:24.112851 24069 consensus_queue.cc:260] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.113992 24068 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630: No bootstrap required, opened a new log
I20240501 20:11:24.114178 24070 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: No bootstrap required, opened a new log
I20240501 20:11:24.114362 24070 ts_tablet_manager.cc:1367] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.001s
I20240501 20:11:24.114610 24069 ts_tablet_manager.cc:1398] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240501 20:11:24.114912 24069 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9: Bootstrap starting.
I20240501 20:11:24.115175 24068 ts_tablet_manager.cc:1367] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.007s
I20240501 20:11:24.115089 24070 raft_consensus.cc:348] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.115281 24070 raft_consensus.cc:374] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:24.115365 24070 raft_consensus.cc:729] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: FOLLOWER
I20240501 20:11:24.115569 24070 consensus_queue.cc:260] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.115891 24068 raft_consensus.cc:348] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.116104 24068 raft_consensus.cc:374] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:24.116159 24070 ts_tablet_manager.cc:1398] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:11:24.116199 24068 raft_consensus.cc:729] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: FOLLOWER
I20240501 20:11:24.116489 24070 tablet_bootstrap.cc:492] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:24.116398 24068 consensus_queue.cc:260] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.116849 24069 tablet_bootstrap.cc:654] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:24.116904 24068 ts_tablet_manager.cc:1398] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:11:24.117162 24068 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:24.118523 24070 tablet_bootstrap.cc:654] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:24.118667 24068 tablet_bootstrap.cc:654] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:24.123014 24069 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9: No bootstrap required, opened a new log
I20240501 20:11:24.123145 24069 ts_tablet_manager.cc:1367] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.007s
I20240501 20:11:24.123301 24070 tablet_bootstrap.cc:492] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa: No bootstrap required, opened a new log
I20240501 20:11:24.123422 24070 ts_tablet_manager.cc:1367] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240501 20:11:24.123883 24068 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630: No bootstrap required, opened a new log
I20240501 20:11:24.123819 24069 raft_consensus.cc:348] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.123921 24070 raft_consensus.cc:348] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.124013 24069 raft_consensus.cc:374] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:24.124024 24068 ts_tablet_manager.cc:1367] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.002s
I20240501 20:11:24.124085 24070 raft_consensus.cc:374] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:24.124081 24069 raft_consensus.cc:729] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Initialized, Role: FOLLOWER
I20240501 20:11:24.124178 24070 raft_consensus.cc:729] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: FOLLOWER
I20240501 20:11:24.124307 24069 consensus_queue.cc:260] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.124370 24070 consensus_queue.cc:260] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.124688 24068 raft_consensus.cc:348] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.124884 24069 ts_tablet_manager.cc:1398] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:11:24.124881 24070 ts_tablet_manager.cc:1398] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240501 20:11:24.124883 24068 raft_consensus.cc:374] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:24.124953 24068 raft_consensus.cc:729] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: FOLLOWER
I20240501 20:11:24.125155 24069 tablet_bootstrap.cc:492] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9: Bootstrap starting.
I20240501 20:11:24.125162 24068 consensus_queue.cc:260] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.125633 24068 ts_tablet_manager.cc:1398] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:11:24.125927 24068 tablet_bootstrap.cc:492] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:24.126539 24069 tablet_bootstrap.cc:654] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:24.127290 24068 tablet_bootstrap.cc:654] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:24.128286 24069 tablet_bootstrap.cc:492] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9: No bootstrap required, opened a new log
I20240501 20:11:24.128414 24069 ts_tablet_manager.cc:1367] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:11:24.129048 24069 raft_consensus.cc:348] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.129244 24069 raft_consensus.cc:374] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:24.129333 24069 raft_consensus.cc:729] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb07a45c9604747986a01ae6f7f7bc9, State: Initialized, Role: FOLLOWER
I20240501 20:11:24.129530 24069 consensus_queue.cc:260] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.130088 24069 ts_tablet_manager.cc:1398] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:11:24.130995 24068 tablet_bootstrap.cc:492] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630: No bootstrap required, opened a new log
I20240501 20:11:24.131130 24068 ts_tablet_manager.cc:1367] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 20:11:24.131819 24068 raft_consensus.cc:348] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.131999 24068 raft_consensus.cc:374] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:24.132086 24068 raft_consensus.cc:729] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: FOLLOWER
I20240501 20:11:24.132285 24068 consensus_queue.cc:260] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.132768 24068 ts_tablet_manager.cc:1398] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:11:24.135569 23987 raft_consensus.cc:482] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:24.135710 23987 raft_consensus.cc:504] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.136183 23987 leader_election.cc:288] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:24.136546 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630" is_pre_election: true
I20240501 20:11:24.136610 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" is_pre_election: true
I20240501 20:11:24.136705 23157 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:24.136795 23432 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:24.137101 23226 leader_election.cc:302] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, a4b53d88de2b4e938f65b317b60b0630; no voters: 
I20240501 20:11:24.137382 23987 raft_consensus.cc:2781] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:24.137459 23987 raft_consensus.cc:482] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:24.137547 23987 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:24.139292 23987 raft_consensus.cc:504] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.139869 23987 leader_election.cc:288] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Requested vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:24.140127 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:24.140275 23157 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:24.140376 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
I20240501 20:11:24.140524 23432 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:24.142926 23157 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:24.143070 23432 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:24.143290 23226 leader_election.cc:302] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, a4b53d88de2b4e938f65b317b60b0630; no voters: 
I20240501 20:11:24.143522 23987 raft_consensus.cc:2781] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:24.143647 23987 raft_consensus.cc:686] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Becoming Leader. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Running, Role: LEADER
I20240501 20:11:24.143850 23987 consensus_queue.cc:237] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.146049 23012 catalog_manager.cc:5484] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 7abd52761123466ca399914a88fb23aa (127.8.68.194). New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:24.224700 23987 raft_consensus.cc:482] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:24.224846 23987 raft_consensus.cc:504] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.225399 23987 leader_election.cc:288] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:24.225888 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630" is_pre_election: true
I20240501 20:11:24.225888 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" is_pre_election: true
I20240501 20:11:24.226121 23157 raft_consensus.cc:2454] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:24.226136 23432 raft_consensus.cc:2454] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:24.226508 23224 leader_election.cc:302] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:24.226547 23987 raft_consensus.cc:482] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:24.226830 24058 raft_consensus.cc:2781] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:24.226751 23987 raft_consensus.cc:504] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.226943 24058 raft_consensus.cc:482] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:24.227084 24058 raft_consensus.cc:3037] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:24.227501 23987 leader_election.cc:288] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:24.227730 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155d934dae348c99ffb489cefcd64b4" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630" is_pre_election: true
I20240501 20:11:24.227936 23157 raft_consensus.cc:2454] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:24.227933 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155d934dae348c99ffb489cefcd64b4" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" is_pre_election: true
I20240501 20:11:24.228173 23432 raft_consensus.cc:2454] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 0.
I20240501 20:11:24.228268 23226 leader_election.cc:302] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, a4b53d88de2b4e938f65b317b60b0630; no voters: 
I20240501 20:11:24.228565 23987 raft_consensus.cc:2781] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:24.228659 23987 raft_consensus.cc:482] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:24.228773 23987 raft_consensus.cc:3037] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:24.229792 24058 raft_consensus.cc:504] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.230381 24058 leader_election.cc:288] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Requested vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:24.230734 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:24.230813 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
I20240501 20:11:24.230916 23157 raft_consensus.cc:3037] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:24.230809 23987 raft_consensus.cc:504] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.230998 23432 raft_consensus.cc:3037] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:24.231523 23987 leader_election.cc:288] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Requested vote from peers a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:24.231726 23156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155d934dae348c99ffb489cefcd64b4" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:24.231740 23431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155d934dae348c99ffb489cefcd64b4" candidate_uuid: "7abd52761123466ca399914a88fb23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
I20240501 20:11:24.231864 23156 raft_consensus.cc:3037] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:24.231936 23431 raft_consensus.cc:3037] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:24.233633 23156 raft_consensus.cc:2454] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:24.233697 23431 raft_consensus.cc:2454] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:24.233987 23226 leader_election.cc:302] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, a4b53d88de2b4e938f65b317b60b0630; no voters: 
I20240501 20:11:24.234346 23987 raft_consensus.cc:2781] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:24.234519 23987 raft_consensus.cc:686] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Becoming Leader. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Running, Role: LEADER
I20240501 20:11:24.234769 23987 consensus_queue.cc:237] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.235661 23157 raft_consensus.cc:2454] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:24.236068 23226 leader_election.cc:302] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7abd52761123466ca399914a88fb23aa, a4b53d88de2b4e938f65b317b60b0630; no voters: 
I20240501 20:11:24.236133 23432 raft_consensus.cc:2454] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7abd52761123466ca399914a88fb23aa in term 1.
I20240501 20:11:24.236369 23987 raft_consensus.cc:2781] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:24.236554 23987 raft_consensus.cc:686] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Becoming Leader. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Running, Role: LEADER
I20240501 20:11:24.236835 23987 consensus_queue.cc:237] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } }
I20240501 20:11:24.237941 23012 catalog_manager.cc:5484] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 7abd52761123466ca399914a88fb23aa (127.8.68.194). New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:24.242240 23012 catalog_manager.cc:5484] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 7abd52761123466ca399914a88fb23aa (127.8.68.194). New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:24.588902 23987 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:24.597409 24058 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:24.603893 24083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:24.604144 24083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:24.611258 24083 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 20:11:24.699306 24058 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:24.713035 24058 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:24.736367 24058 consensus_queue.cc:1035] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:24.748816 24058 consensus_queue.cc:1035] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 206.448 ms
  time per row: 0.103224 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:11:25.827649  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:33701
--local_ip_for_outbound_sockets=127.8.68.196
--tserver_master_addrs=127.8.68.254:46623
--webserver_port=33859
--webserver_interface=127.8.68.196
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:26.141464 24138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:26.141711 24138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:26.141845 24138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:26.148804 24138 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:26.148923 24138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:26.149065 24138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:11:26.157352 24138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:33701
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=33859
--tserver_master_addrs=127.8.68.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:26.158460 24138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:26.159996 24138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:26.170996 24143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:26.171422 24145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:26.171984 24147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:26.172443 24138 server_base.cc:1034] running on GCE node
I20240501 20:11:26.173053 24138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:26.173946 24138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:26.175163 24138 hybrid_clock.cc:648] HybridClock initialized: now 1714594286175095 us; error 46 us; skew 500 ppm
I20240501 20:11:26.175491 24138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:26.180889 24138 webserver.cc:460] Webserver started at http://127.8.68.196:33859/ using document root <none> and password file <none>
I20240501 20:11:26.181545 24138 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:26.181669 24138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:26.186481 24138 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:11:26.189013 24153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:26.190181 24138 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:26.190335 24138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5bd8f7fde1784f5fa1ab24113dc27934"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:26.191062 24138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:26.212441 24138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:26.213079 24138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:26.213328 24138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:26.214094 24138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:26.215438 24138 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:26.215512 24138 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:26.215627 24138 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:26.215678 24138 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:26.243170 24138 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:33701
I20240501 20:11:26.243223 24270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:33701 every 8 connection(s)
I20240501 20:11:26.244416 24138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:26.254689  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 24138
I20240501 20:11:26.255467  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.198:43689
--local_ip_for_outbound_sockets=127.8.68.198
--tserver_master_addrs=127.8.68.254:46623
--webserver_port=43189
--webserver_interface=127.8.68.198
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:26.255880 24271 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46623
I20240501 20:11:26.256124 24271 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:26.256835 24271 heartbeater.cc:511] Master 127.8.68.254:46623 requested a full tablet report, sending...
I20240501 20:11:26.258496 23013 ts_manager.cc:194] Re-registered known tserver with Master: 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196:33701)
I20240501 20:11:26.259501 23013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:47491
W20240501 20:11:26.576901 24275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:26.577142 24275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:26.577255 24275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:26.584560 24275 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:26.584663 24275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:26.584807 24275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.198
I20240501 20:11:26.593271 24275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.198:43689
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.68.198
--webserver_port=43189
--tserver_master_addrs=127.8.68.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.198
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:26.594408 24275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:26.596007 24275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:26.606897 24280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:26.607319 24283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:26.607822 24281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:26.608422 24275 server_base.cc:1034] running on GCE node
I20240501 20:11:26.609247 24275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:26.610062 24275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:26.611254 24275 hybrid_clock.cc:648] HybridClock initialized: now 1714594286611200 us; error 53 us; skew 500 ppm
I20240501 20:11:26.611524 24275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:26.613691 24275 webserver.cc:460] Webserver started at http://127.8.68.198:43189/ using document root <none> and password file <none>
I20240501 20:11:26.614542 24275 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:26.614677 24275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:26.619261 24275 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 20:11:26.621639 24289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:26.622776 24275 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:11:26.622925 24275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "527d2c1d7b3d45aeb992646b89911c14"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:26.623689 24275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:26.639494 24275 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:26.640164 24275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:26.640406 24275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:26.641196 24275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:26.642535 24275 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:26.642609 24275 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:26.642721 24275 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:26.642771 24275 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:26.669245 24275 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.198:43689
I20240501 20:11:26.669296 24402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.198:43689 every 8 connection(s)
I20240501 20:11:26.670514 24275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:11:26.674439  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 24275
I20240501 20:11:26.675127  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:33911
--local_ip_for_outbound_sockets=127.8.68.197
--tserver_master_addrs=127.8.68.254:46623
--webserver_port=45739
--webserver_interface=127.8.68.197
--builtin_ntp_servers=127.8.68.212:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:26.682268 24403 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46623
I20240501 20:11:26.682545 24403 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:26.683271 24403 heartbeater.cc:511] Master 127.8.68.254:46623 requested a full tablet report, sending...
I20240501 20:11:26.684556 23013 ts_manager.cc:194] Re-registered known tserver with Master: 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689)
I20240501 20:11:26.685304 23013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.198:36809
W20240501 20:11:26.994582 24406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:26.994853 24406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:26.994957 24406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:27.002110 24406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:27.002213 24406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:27.002357 24406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:11:27.010629 24406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:33911
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=45739
--tserver_master_addrs=127.8.68.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:27.011693 24406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:27.013227 24406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:27.024109 24411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:27.024585 24412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:27.024938 24414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:27.025588 24406 server_base.cc:1034] running on GCE node
I20240501 20:11:27.026109 24406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:27.026948 24406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:27.028159 24406 hybrid_clock.cc:648] HybridClock initialized: now 1714594287028117 us; error 32 us; skew 500 ppm
I20240501 20:11:27.028455 24406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:27.030500 24406 webserver.cc:460] Webserver started at http://127.8.68.197:45739/ using document root <none> and password file <none>
I20240501 20:11:27.031133 24406 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:27.031261 24406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:27.035804 24406 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.003s
I20240501 20:11:27.038699 24420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:27.039861 24406 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240501 20:11:27.039994 24406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c"
format_stamp: "Formatted at 2024-05-01 20:11:17 on dist-test-slave-08p9"
I20240501 20:11:27.040741 24406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:27.073259 24406 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:27.073899 24406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:27.074148 24406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:27.074976 24406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:27.076587 24406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:27.076663 24406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:27.076782 24406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:27.076833 24406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:27.105064 24406 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:33911
I20240501 20:11:27.105127 24533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:33911 every 8 connection(s)
I20240501 20:11:27.106330 24406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:11:27.115000  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 24406
I20240501 20:11:27.117159 24534 heartbeater.cc:347] Connected to a master server at 127.8.68.254:46623
I20240501 20:11:27.117408 24534 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:27.118103 24534 heartbeater.cc:511] Master 127.8.68.254:46623 requested a full tablet report, sending...
I20240501 20:11:27.119393 23013 ts_manager.cc:194] Re-registered known tserver with Master: 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197:33911)
I20240501 20:11:27.120136 23013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:50209
I20240501 20:11:27.261651 24271 heartbeater.cc:503] Master 127.8.68.254:46623 was elected leader, sending a full tablet report...
I20240501 20:11:27.593986 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:27.594252 24338 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:27.600852 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:27.607411 23135 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:27.614718 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:27.616393 23272 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:27.687495 24403 heartbeater.cc:503] Master 127.8.68.254:46623 was elected leader, sending a full tablet report...
I20240501 20:11:27.775738 24338 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:27.776873 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:27.787679 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:27.788506 23272 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:27.831096 23135 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:27.836557 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:28.011909 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.012452 24338 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.023501 23272 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:28.026525 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.033851 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:28.035840 23135 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:28.122144 24534 heartbeater.cc:503] Master 127.8.68.254:46623 was elected leader, sending a full tablet report...
I20240501 20:11:28.184058 24338 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.184763 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.203078 23135 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:28.205739 23272 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:28.209334 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.228124 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:28.295037 23292 consensus_queue.cc:237] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:28.298027 23156 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.298660 24538 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.311832 23432 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.311558 24538 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:28.312686 24687 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.312692 23156 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:28.318859 23006 catalog_manager.cc:5484] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
W20240501 20:11:28.320773 23224 consensus_peers.cc:485] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa -> Peer 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689): Couldn't send request to peer 527d2c1d7b3d45aeb992646b89911c14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d940d4d4f1a54388b49f6794ca57d0df. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:28.322365 23432 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:28.352962 23432 consensus_queue.cc:237] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:28.355604 23156 raft_consensus.cc:1261] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.355855 23292 raft_consensus.cc:1261] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.356402 24702 consensus_queue.cc:1035] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.356803 24046 consensus_queue.cc:1035] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.360455 23156 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.359856 24702 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.361711 23292 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
W20240501 20:11:28.364753 23358 consensus_peers.cc:485] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 -> Peer 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196:33701): Couldn't send request to peer 5bd8f7fde1784f5fa1ab24113dc27934. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bb01b1a8a02483e8d49cf2327886f18. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:28.365581 23006 catalog_manager.cc:5484] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.378083 23292 consensus_queue.cc:237] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } }
I20240501 20:11:28.380312 23156 raft_consensus.cc:1261] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.380556 23432 raft_consensus.cc:1261] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.381096 24687 consensus_queue.cc:1035] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.381464 24538 consensus_queue.cc:1035] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.384240 24691 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:28.384788 23432 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
W20240501 20:11:28.387411 23224 consensus_peers.cc:485] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa -> Peer 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197:33911): Couldn't send request to peer 8d0f4e53cd9e469c9d5dffee5c42ea6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc9d141aa6ef4fa190ae8b9e7f4a4ec8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:28.389037 23156 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:28.390147 23006 catalog_manager.cc:5484] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:28.403738 23156 consensus_queue.cc:237] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:28.406040 23292 raft_consensus.cc:1261] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.406637 24720 consensus_queue.cc:1035] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.407069 23432 raft_consensus.cc:1261] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.407599 24720 consensus_queue.cc:1035] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.409703 23977 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:28.410387 23292 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
W20240501 20:11:28.413612 23090 consensus_peers.cc:485] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 -> Peer 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689): Couldn't send request to peer 527d2c1d7b3d45aeb992646b89911c14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e0b80ffdd2d4d1c8f6d6c106c724efa. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:28.414593 23013 catalog_manager.cc:5484] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "a4b53d88de2b4e938f65b317b60b0630" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:28.418918 23432 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:28.436852 23432 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:28.439188 23156 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.439404 23292 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.439699 24733 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:28.440125 23358 consensus_peers.cc:485] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 -> Peer 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196:33701): Couldn't send request to peer 5bd8f7fde1784f5fa1ab24113dc27934. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c56eb0a04104909ae46e1c7c8d732c0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:28.440165 24702 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.442922 24046 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.443584 23292 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.445861 23156 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.447775 23013 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.460444 23292 consensus_queue.cc:237] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } }
I20240501 20:11:28.462350 23156 raft_consensus.cc:1261] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.462517 23432 raft_consensus.cc:1261] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:28.462780 23224 consensus_peers.cc:485] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa -> Peer 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197:33911): Couldn't send request to peer 8d0f4e53cd9e469c9d5dffee5c42ea6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6155d934dae348c99ffb489cefcd64b4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:28.463085 24687 consensus_queue.cc:1035] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.463351 24538 consensus_queue.cc:1035] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.465770 24538 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:28.466337 23432 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:28.466506 23156 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:28.469839 23013 catalog_manager.cc:5484] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index -1 to 4, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:28.482547 23292 consensus_queue.cc:237] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:28.484566 23156 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:28.484980 23224 consensus_peers.cc:485] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa -> Peer 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689): Couldn't send request to peer 527d2c1d7b3d45aeb992646b89911c14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d3d2792048e4b4aa59c034e0539ddb7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:28.485044 23432 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.485260 24687 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.485560 24538 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.488219 24687 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:28.491086 23432 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:28.492413 23011 catalog_manager.cc:5484] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:28.503753 23156 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:28.510869 23292 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:28.512733 23156 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.513239 24691 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.513358 23432 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:28.514024 24691 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:28.516211 24687 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.516850 23156 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
W20240501 20:11:28.519101 23224 consensus_peers.cc:485] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa -> Peer 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196:33701): Couldn't send request to peer 5bd8f7fde1784f5fa1ab24113dc27934. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7475f202d7f2476fbd69ef838d7e1ee2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:28.519573 23432 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.520514 23011 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index -1 to 4, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:28.760046 24800 ts_tablet_manager.cc:902] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: Initiating tablet copy from peer 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:28.761783 24800 tablet_copy_client.cc:308] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:41175
I20240501 20:11:28.767760 23313 tablet_copy_service.cc:140] P 7abd52761123466ca399914a88fb23aa: Received BeginTabletCopySession request for tablet d940d4d4f1a54388b49f6794ca57d0df from peer 527d2c1d7b3d45aeb992646b89911c14 ({username='slave'} at 127.8.68.198:47231)
I20240501 20:11:28.767969 23313 tablet_copy_service.cc:161] P 7abd52761123466ca399914a88fb23aa: Beginning new tablet copy session on tablet d940d4d4f1a54388b49f6794ca57d0df from peer 527d2c1d7b3d45aeb992646b89911c14 at {username='slave'} at 127.8.68.198:47231: session id = 527d2c1d7b3d45aeb992646b89911c14-d940d4d4f1a54388b49f6794ca57d0df
I20240501 20:11:28.770591 23313 tablet_copy_source_session.cc:205] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:28.774880 24800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940d4d4f1a54388b49f6794ca57d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.784399 24800 tablet_copy_client.cc:783] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:28.784667 24800 tablet_copy_client.cc:647] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:28.786744 24800 tablet_copy_client.cc:523] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:28.789523 24800 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap starting.
I20240501 20:11:28.798667 24800 log.cc:824] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:28.838685 24800 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:28.839248 24800 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap complete.
I20240501 20:11:28.839743 24800 ts_tablet_manager.cc:1367] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: Time spent bootstrapping tablet: real 0.050s	user 0.047s	sys 0.004s
I20240501 20:11:28.845607 24800 raft_consensus.cc:348] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:28.846050 24800 raft_consensus.cc:729] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 527d2c1d7b3d45aeb992646b89911c14, State: Initialized, Role: LEARNER
I20240501 20:11:28.846616 24800 consensus_queue.cc:260] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:28.854779 24800 ts_tablet_manager.cc:1398] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: Time spent starting tablet: real 0.015s	user 0.015s	sys 0.000s
I20240501 20:11:28.856529 23313 tablet_copy_service.cc:342] P 7abd52761123466ca399914a88fb23aa: Request end of tablet copy session 527d2c1d7b3d45aeb992646b89911c14-d940d4d4f1a54388b49f6794ca57d0df received from {username='slave'} at 127.8.68.198:47231
I20240501 20:11:28.856689 23313 tablet_copy_service.cc:433] P 7abd52761123466ca399914a88fb23aa: ending tablet copy session 527d2c1d7b3d45aeb992646b89911c14-d940d4d4f1a54388b49f6794ca57d0df on tablet d940d4d4f1a54388b49f6794ca57d0df with peer 527d2c1d7b3d45aeb992646b89911c14
I20240501 20:11:28.869351 24845 ts_tablet_manager.cc:902] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Initiating tablet copy from peer 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:28.889967 24845 tablet_copy_client.cc:308] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:41175
I20240501 20:11:28.895929 23313 tablet_copy_service.cc:140] P 7abd52761123466ca399914a88fb23aa: Received BeginTabletCopySession request for tablet 6155d934dae348c99ffb489cefcd64b4 from peer 8d0f4e53cd9e469c9d5dffee5c42ea6c ({username='slave'} at 127.8.68.197:60865)
I20240501 20:11:28.896099 23313 tablet_copy_service.cc:161] P 7abd52761123466ca399914a88fb23aa: Beginning new tablet copy session on tablet 6155d934dae348c99ffb489cefcd64b4 from peer 8d0f4e53cd9e469c9d5dffee5c42ea6c at {username='slave'} at 127.8.68.197:60865: session id = 8d0f4e53cd9e469c9d5dffee5c42ea6c-6155d934dae348c99ffb489cefcd64b4
I20240501 20:11:28.897948 23313 tablet_copy_source_session.cc:205] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:28.901692 24845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155d934dae348c99ffb489cefcd64b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.911435 24845 tablet_copy_client.cc:783] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:28.911918 24845 tablet_copy_client.cc:647] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:28.912221 24856 ts_tablet_manager.cc:902] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Initiating tablet copy from peer 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:28.914135 24856 tablet_copy_client.cc:308] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:41175
I20240501 20:11:28.914212 24845 tablet_copy_client.cc:523] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:28.914814 23313 tablet_copy_service.cc:140] P 7abd52761123466ca399914a88fb23aa: Received BeginTabletCopySession request for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8 from peer 8d0f4e53cd9e469c9d5dffee5c42ea6c ({username='slave'} at 127.8.68.197:60865)
I20240501 20:11:28.914950 23313 tablet_copy_service.cc:161] P 7abd52761123466ca399914a88fb23aa: Beginning new tablet copy session on tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8 from peer 8d0f4e53cd9e469c9d5dffee5c42ea6c at {username='slave'} at 127.8.68.197:60865: session id = 8d0f4e53cd9e469c9d5dffee5c42ea6c-dc9d141aa6ef4fa190ae8b9e7f4a4ec8
I20240501 20:11:28.917709 23313 tablet_copy_source_session.cc:205] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:28.918107 24845 tablet_bootstrap.cc:492] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Bootstrap starting.
I20240501 20:11:28.918769 24856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.924471 24856 tablet_copy_client.cc:783] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:28.924716 24856 tablet_copy_client.cc:647] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:28.926189 24856 tablet_copy_client.cc:523] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:28.929632 24856 tablet_bootstrap.cc:492] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Bootstrap starting.
I20240501 20:11:28.930513 24845 log.cc:824] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:28.937983 24800 ts_tablet_manager.cc:902] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: Initiating tablet copy from peer a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789)
I20240501 20:11:28.942615 24800 tablet_copy_client.cc:308] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:39789
I20240501 20:11:28.949597 23177 tablet_copy_service.cc:140] P a4b53d88de2b4e938f65b317b60b0630: Received BeginTabletCopySession request for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa from peer 527d2c1d7b3d45aeb992646b89911c14 ({username='slave'} at 127.8.68.198:34399)
I20240501 20:11:28.949846 23177 tablet_copy_service.cc:161] P a4b53d88de2b4e938f65b317b60b0630: Beginning new tablet copy session on tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa from peer 527d2c1d7b3d45aeb992646b89911c14 at {username='slave'} at 127.8.68.198:34399: session id = 527d2c1d7b3d45aeb992646b89911c14-0e0b80ffdd2d4d1c8f6d6c106c724efa
I20240501 20:11:28.953456 23177 tablet_copy_source_session.cc:205] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:28.955165 24800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:28.961172 24800 tablet_copy_client.cc:783] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:28.961405 24800 tablet_copy_client.cc:647] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:28.963300 24800 tablet_copy_client.cc:523] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:28.966853 24800 tablet_bootstrap.cc:492] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap starting.
I20240501 20:11:28.969815 24856 tablet_bootstrap.cc:492] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:28.970366 24856 tablet_bootstrap.cc:492] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Bootstrap complete.
I20240501 20:11:28.970759 24856 ts_tablet_manager.cc:1367] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Time spent bootstrapping tablet: real 0.041s	user 0.039s	sys 0.004s
I20240501 20:11:28.972339 24845 tablet_bootstrap.cc:492] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:28.972817 24845 tablet_bootstrap.cc:492] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Bootstrap complete.
I20240501 20:11:28.973141 24845 ts_tablet_manager.cc:1367] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Time spent bootstrapping tablet: real 0.055s	user 0.054s	sys 0.003s
I20240501 20:11:28.977447 24845 raft_consensus.cc:348] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } }
I20240501 20:11:28.977962 24845 raft_consensus.cc:729] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d0f4e53cd9e469c9d5dffee5c42ea6c, State: Initialized, Role: LEARNER
I20240501 20:11:28.978240 24868 ts_tablet_manager.cc:902] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: Initiating tablet copy from peer ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:28.978534 24845 consensus_queue.cc:260] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } }
I20240501 20:11:28.980485 24845 ts_tablet_manager.cc:1398] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20240501 20:11:28.977437 24856 raft_consensus.cc:348] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } }
I20240501 20:11:28.981463 24856 raft_consensus.cc:729] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d0f4e53cd9e469c9d5dffee5c42ea6c, State: Initialized, Role: LEARNER
I20240501 20:11:28.981665 24856 consensus_queue.cc:260] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } }
I20240501 20:11:28.982236 23313 tablet_copy_service.cc:342] P 7abd52761123466ca399914a88fb23aa: Request end of tablet copy session 8d0f4e53cd9e469c9d5dffee5c42ea6c-6155d934dae348c99ffb489cefcd64b4 received from {username='slave'} at 127.8.68.197:60865
I20240501 20:11:28.982425 23313 tablet_copy_service.cc:433] P 7abd52761123466ca399914a88fb23aa: ending tablet copy session 8d0f4e53cd9e469c9d5dffee5c42ea6c-6155d934dae348c99ffb489cefcd64b4 on tablet 6155d934dae348c99ffb489cefcd64b4 with peer 8d0f4e53cd9e469c9d5dffee5c42ea6c
I20240501 20:11:28.993518 24856 ts_tablet_manager.cc:1398] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Time spent starting tablet: real 0.023s	user 0.011s	sys 0.008s
I20240501 20:11:28.995034 23313 tablet_copy_service.cc:342] P 7abd52761123466ca399914a88fb23aa: Request end of tablet copy session 8d0f4e53cd9e469c9d5dffee5c42ea6c-dc9d141aa6ef4fa190ae8b9e7f4a4ec8 received from {username='slave'} at 127.8.68.197:60865
I20240501 20:11:28.995188 23313 tablet_copy_service.cc:433] P 7abd52761123466ca399914a88fb23aa: ending tablet copy session 8d0f4e53cd9e469c9d5dffee5c42ea6c-dc9d141aa6ef4fa190ae8b9e7f4a4ec8 on tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8 with peer 8d0f4e53cd9e469c9d5dffee5c42ea6c
I20240501 20:11:28.998785 24868 tablet_copy_client.cc:308] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:45603
I20240501 20:11:28.999154 24876 ts_tablet_manager.cc:902] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: Initiating tablet copy from peer 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:29.003674 24876 tablet_copy_client.cc:308] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:41175
I20240501 20:11:29.008054 23452 tablet_copy_service.cc:140] P ebb07a45c9604747986a01ae6f7f7bc9: Received BeginTabletCopySession request for tablet 6bb01b1a8a02483e8d49cf2327886f18 from peer 5bd8f7fde1784f5fa1ab24113dc27934 ({username='slave'} at 127.8.68.196:59639)
I20240501 20:11:29.008278 23452 tablet_copy_service.cc:161] P ebb07a45c9604747986a01ae6f7f7bc9: Beginning new tablet copy session on tablet 6bb01b1a8a02483e8d49cf2327886f18 from peer 5bd8f7fde1784f5fa1ab24113dc27934 at {username='slave'} at 127.8.68.196:59639: session id = 5bd8f7fde1784f5fa1ab24113dc27934-6bb01b1a8a02483e8d49cf2327886f18
I20240501 20:11:29.010677 23313 tablet_copy_service.cc:140] P 7abd52761123466ca399914a88fb23aa: Received BeginTabletCopySession request for tablet 7475f202d7f2476fbd69ef838d7e1ee2 from peer 5bd8f7fde1784f5fa1ab24113dc27934 ({username='slave'} at 127.8.68.196:53005)
I20240501 20:11:29.010840 23313 tablet_copy_service.cc:161] P 7abd52761123466ca399914a88fb23aa: Beginning new tablet copy session on tablet 7475f202d7f2476fbd69ef838d7e1ee2 from peer 5bd8f7fde1784f5fa1ab24113dc27934 at {username='slave'} at 127.8.68.196:53005: session id = 5bd8f7fde1784f5fa1ab24113dc27934-7475f202d7f2476fbd69ef838d7e1ee2
I20240501 20:11:29.011857 23452 tablet_copy_source_session.cc:205] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:29.012842 23313 tablet_copy_source_session.cc:205] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:29.014237 24800 tablet_bootstrap.cc:492] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:29.014588 24800 tablet_bootstrap.cc:492] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap complete.
I20240501 20:11:29.014830 24800 ts_tablet_manager.cc:1367] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: Time spent bootstrapping tablet: real 0.048s	user 0.036s	sys 0.009s
I20240501 20:11:29.015617 24800 raft_consensus.cc:348] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:29.015890 24800 raft_consensus.cc:729] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 527d2c1d7b3d45aeb992646b89911c14, State: Initialized, Role: LEARNER
I20240501 20:11:29.016099 24800 consensus_queue.cc:260] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:29.016862 24868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb01b1a8a02483e8d49cf2327886f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:29.017688 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7475f202d7f2476fbd69ef838d7e1ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:29.018968 24800 ts_tablet_manager.cc:1398] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240501 20:11:29.019788 23177 tablet_copy_service.cc:342] P a4b53d88de2b4e938f65b317b60b0630: Request end of tablet copy session 527d2c1d7b3d45aeb992646b89911c14-0e0b80ffdd2d4d1c8f6d6c106c724efa received from {username='slave'} at 127.8.68.198:34399
I20240501 20:11:29.019913 23177 tablet_copy_service.cc:433] P a4b53d88de2b4e938f65b317b60b0630: ending tablet copy session 527d2c1d7b3d45aeb992646b89911c14-0e0b80ffdd2d4d1c8f6d6c106c724efa on tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa with peer 527d2c1d7b3d45aeb992646b89911c14
I20240501 20:11:29.028020 24868 tablet_copy_client.cc:783] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:29.028091 24883 ts_tablet_manager.cc:902] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: Initiating tablet copy from peer ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:29.028069 24876 tablet_copy_client.cc:783] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:29.028322 24868 tablet_copy_client.cc:647] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:29.028453 24876 tablet_copy_client.cc:647] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:29.030256 24876 tablet_copy_client.cc:523] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:29.030706 24868 tablet_copy_client.cc:523] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:29.033336 24876 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap starting.
I20240501 20:11:29.034225 24868 tablet_bootstrap.cc:492] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap starting.
I20240501 20:11:29.035401 24883 tablet_copy_client.cc:308] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:45603
I20240501 20:11:29.036080 23452 tablet_copy_service.cc:140] P ebb07a45c9604747986a01ae6f7f7bc9: Received BeginTabletCopySession request for tablet 6c56eb0a04104909ae46e1c7c8d732c0 from peer 5bd8f7fde1784f5fa1ab24113dc27934 ({username='slave'} at 127.8.68.196:59639)
I20240501 20:11:29.036247 23452 tablet_copy_service.cc:161] P ebb07a45c9604747986a01ae6f7f7bc9: Beginning new tablet copy session on tablet 6c56eb0a04104909ae46e1c7c8d732c0 from peer 5bd8f7fde1784f5fa1ab24113dc27934 at {username='slave'} at 127.8.68.196:59639: session id = 5bd8f7fde1784f5fa1ab24113dc27934-6c56eb0a04104909ae46e1c7c8d732c0
I20240501 20:11:29.038769 23452 tablet_copy_source_session.cc:205] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:29.040982 24883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c56eb0a04104909ae46e1c7c8d732c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:29.043651 24876 log.cc:824] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:29.047111 24883 tablet_copy_client.cc:783] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:29.047339 24883 tablet_copy_client.cc:647] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:29.049000 24883 tablet_copy_client.cc:523] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:29.052309 24883 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap starting.
I20240501 20:11:29.060087 24800 ts_tablet_manager.cc:902] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: Initiating tablet copy from peer 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:29.070559 24800 tablet_copy_client.cc:308] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:41175
I20240501 20:11:29.071199 23313 tablet_copy_service.cc:140] P 7abd52761123466ca399914a88fb23aa: Received BeginTabletCopySession request for tablet 3d3d2792048e4b4aa59c034e0539ddb7 from peer 527d2c1d7b3d45aeb992646b89911c14 ({username='slave'} at 127.8.68.198:47231)
I20240501 20:11:29.071341 23313 tablet_copy_service.cc:161] P 7abd52761123466ca399914a88fb23aa: Beginning new tablet copy session on tablet 3d3d2792048e4b4aa59c034e0539ddb7 from peer 527d2c1d7b3d45aeb992646b89911c14 at {username='slave'} at 127.8.68.198:47231: session id = 527d2c1d7b3d45aeb992646b89911c14-3d3d2792048e4b4aa59c034e0539ddb7
I20240501 20:11:29.073968 23313 tablet_copy_source_session.cc:205] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:29.075547 24800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3d2792048e4b4aa59c034e0539ddb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:29.083544 24800 tablet_copy_client.cc:783] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:29.083801 24800 tablet_copy_client.cc:647] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:29.085918 24876 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:29.086555 24876 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap complete.
I20240501 20:11:29.087106 24876 ts_tablet_manager.cc:1367] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: Time spent bootstrapping tablet: real 0.054s	user 0.049s	sys 0.005s
I20240501 20:11:29.092310 24868 tablet_bootstrap.cc:492] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:29.092641 24800 tablet_copy_client.cc:523] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:29.092765 24868 tablet_bootstrap.cc:492] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap complete.
I20240501 20:11:29.093025 24868 ts_tablet_manager.cc:1367] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: Time spent bootstrapping tablet: real 0.059s	user 0.048s	sys 0.004s
I20240501 20:11:29.094422 24868 raft_consensus.cc:348] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:29.095002 24868 raft_consensus.cc:729] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5bd8f7fde1784f5fa1ab24113dc27934, State: Initialized, Role: LEARNER
I20240501 20:11:29.095528 24868 consensus_queue.cc:260] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:29.094430 24876 raft_consensus.cc:348] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:29.096169 24876 raft_consensus.cc:729] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5bd8f7fde1784f5fa1ab24113dc27934, State: Initialized, Role: LEARNER
I20240501 20:11:29.096403 24876 consensus_queue.cc:260] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:29.097869 24800 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap starting.
I20240501 20:11:29.101068 24868 ts_tablet_manager.cc:1398] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20240501 20:11:29.101549 24876 ts_tablet_manager.cc:1398] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.003s
I20240501 20:11:29.101768 24883 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:29.102140 24883 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap complete.
I20240501 20:11:29.102399 24883 ts_tablet_manager.cc:1367] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: Time spent bootstrapping tablet: real 0.050s	user 0.039s	sys 0.012s
I20240501 20:11:29.102900 23313 tablet_copy_service.cc:342] P 7abd52761123466ca399914a88fb23aa: Request end of tablet copy session 5bd8f7fde1784f5fa1ab24113dc27934-7475f202d7f2476fbd69ef838d7e1ee2 received from {username='slave'} at 127.8.68.196:53005
I20240501 20:11:29.103031 23313 tablet_copy_service.cc:433] P 7abd52761123466ca399914a88fb23aa: ending tablet copy session 5bd8f7fde1784f5fa1ab24113dc27934-7475f202d7f2476fbd69ef838d7e1ee2 on tablet 7475f202d7f2476fbd69ef838d7e1ee2 with peer 5bd8f7fde1784f5fa1ab24113dc27934
I20240501 20:11:29.103089 24883 raft_consensus.cc:348] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:29.103303 24883 raft_consensus.cc:729] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5bd8f7fde1784f5fa1ab24113dc27934, State: Initialized, Role: LEARNER
I20240501 20:11:29.103473 24883 consensus_queue.cc:260] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:29.103942 24883 ts_tablet_manager.cc:1398] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240501 20:11:29.105044 23452 tablet_copy_service.cc:342] P ebb07a45c9604747986a01ae6f7f7bc9: Request end of tablet copy session 5bd8f7fde1784f5fa1ab24113dc27934-6bb01b1a8a02483e8d49cf2327886f18 received from {username='slave'} at 127.8.68.196:59639
I20240501 20:11:29.105191 23452 tablet_copy_service.cc:433] P ebb07a45c9604747986a01ae6f7f7bc9: ending tablet copy session 5bd8f7fde1784f5fa1ab24113dc27934-6bb01b1a8a02483e8d49cf2327886f18 on tablet 6bb01b1a8a02483e8d49cf2327886f18 with peer 5bd8f7fde1784f5fa1ab24113dc27934
I20240501 20:11:29.105665 23451 tablet_copy_service.cc:342] P ebb07a45c9604747986a01ae6f7f7bc9: Request end of tablet copy session 5bd8f7fde1784f5fa1ab24113dc27934-6c56eb0a04104909ae46e1c7c8d732c0 received from {username='slave'} at 127.8.68.196:59639
I20240501 20:11:29.105783 23451 tablet_copy_service.cc:433] P ebb07a45c9604747986a01ae6f7f7bc9: ending tablet copy session 5bd8f7fde1784f5fa1ab24113dc27934-6c56eb0a04104909ae46e1c7c8d732c0 on tablet 6c56eb0a04104909ae46e1c7c8d732c0 with peer 5bd8f7fde1784f5fa1ab24113dc27934
I20240501 20:11:29.139729 24800 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:29.140079 24800 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap complete.
I20240501 20:11:29.140282 24800 ts_tablet_manager.cc:1367] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: Time spent bootstrapping tablet: real 0.043s	user 0.034s	sys 0.004s
I20240501 20:11:29.140885 24800 raft_consensus.cc:348] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:29.141052 24800 raft_consensus.cc:729] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 527d2c1d7b3d45aeb992646b89911c14, State: Initialized, Role: LEARNER
I20240501 20:11:29.141238 24800 consensus_queue.cc:260] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:29.141691 24800 ts_tablet_manager.cc:1398] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:29.142386 23313 tablet_copy_service.cc:342] P 7abd52761123466ca399914a88fb23aa: Request end of tablet copy session 527d2c1d7b3d45aeb992646b89911c14-3d3d2792048e4b4aa59c034e0539ddb7 received from {username='slave'} at 127.8.68.198:47231
I20240501 20:11:29.142520 23313 tablet_copy_service.cc:433] P 7abd52761123466ca399914a88fb23aa: ending tablet copy session 527d2c1d7b3d45aeb992646b89911c14-3d3d2792048e4b4aa59c034e0539ddb7 on tablet 3d3d2792048e4b4aa59c034e0539ddb7 with peer 527d2c1d7b3d45aeb992646b89911c14
I20240501 20:11:29.234154 24358 raft_consensus.cc:1203] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:29.314755 24489 raft_consensus.cc:1203] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:29.336427 24489 raft_consensus.cc:1203] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:29.364950 24225 raft_consensus.cc:1203] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:29.392755 24358 raft_consensus.cc:1203] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:29.407179 24225 raft_consensus.cc:1203] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:29.589965 24225 raft_consensus.cc:1203] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:29.621461 24939 raft_consensus.cc:1050] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: attempting to promote NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 to VOTER
I20240501 20:11:29.622260 24939 consensus_queue.cc:237] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:29.624413 23431 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.624476 23156 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.624707 24358 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.624951 24940 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.625291 24939 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.625553 24987 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.628646 24687 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.629249 24357 raft_consensus.cc:1203] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:29.633240 23156 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.633864 23431 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.637507 24357 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.638236 23006 catalog_manager.cc:5484] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.686462 23292 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df"
dest_uuid: "7abd52761123466ca399914a88fb23aa"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39134
I20240501 20:11:29.686686 23292 raft_consensus.cc:595] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:29.724257 24987 raft_consensus.cc:1050] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa: attempting to promote NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c to VOTER
I20240501 20:11:29.724984 24987 consensus_queue.cc:237] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } }
I20240501 20:11:29.727358 23156 raft_consensus.cc:1261] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.727874 24489 raft_consensus.cc:1261] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.727896 24940 consensus_queue.cc:1035] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.728528 24940 consensus_queue.cc:1035] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.730131 23431 raft_consensus.cc:1261] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.731664 24987 consensus_queue.cc:1035] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.734690 24987 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.738888 24489 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.745157 23013 catalog_manager.cc:5484] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c reported cstate change: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.746531 23156 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.752415 23432 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.756315 23292 consensus_queue.cc:237] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } }
I20240501 20:11:29.758260 23156 raft_consensus.cc:1261] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.758643 24489 raft_consensus.cc:1261] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.759567 24939 consensus_queue.cc:1035] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.759920 24987 consensus_queue.cc:1035] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.761962 24987 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.762687 23156 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.765542 23002 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:29.766052 24489 raft_consensus.cc:2932] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.766614 23013 catalog_manager.cc:5484] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index 5 to 6, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:29.770735 23409 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8 with delete_type TABLET_DATA_TOMBSTONED (TS ebb07a45c9604747986a01ae6f7f7bc9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60092
I20240501 20:11:29.771461 25046 tablet_replica.cc:330] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9: stopping tablet replica
I20240501 20:11:29.771814 25046 raft_consensus.cc:2229] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:29.772059 25046 raft_consensus.cc:2258] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:29.773473 25046 ts_tablet_manager.cc:1875] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:29.776041 24941 raft_consensus.cc:1050] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9: attempting to promote NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 to VOTER
I20240501 20:11:29.777103 24941 consensus_queue.cc:237] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:29.778503 25046 ts_tablet_manager.cc:1888] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:29.778664 25046 log.cc:1196] T dc9d141aa6ef4fa190ae8b9e7f4a4ec8 P ebb07a45c9604747986a01ae6f7f7bc9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc9d141aa6ef4fa190ae8b9e7f4a4ec8
I20240501 20:11:29.779547 23156 raft_consensus.cc:1261] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.779693 23292 raft_consensus.cc:1261] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.780077 24942 consensus_queue.cc:1035] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.780184 22999 catalog_manager.cc:4830] TS ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603): tablet dc9d141aa6ef4fa190ae8b9e7f4a4ec8 (table rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]) successfully deleted
I20240501 20:11:29.781157 24225 raft_consensus.cc:1261] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.781669 24942 consensus_queue.cc:1035] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.782588 24046 consensus_queue.cc:1035] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.784296 24941 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.784785 23292 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.787305 24225 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.788002 23156 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.791695 23011 catalog_manager.cc:5484] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.796296 23432 consensus_queue.cc:237] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:29.797844 24225 raft_consensus.cc:1261] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.797955 23292 raft_consensus.cc:1261] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.798367 25048 consensus_queue.cc:1035] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.798699 24942 consensus_queue.cc:1035] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.800698 24941 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.801246 24225 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.801297 23292 raft_consensus.cc:2932] T 6bb01b1a8a02483e8d49cf2327886f18 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.804332 22999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6bb01b1a8a02483e8d49cf2327886f18 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:29.804984 23013 catalog_manager.cc:5484] T 6bb01b1a8a02483e8d49cf2327886f18 P 5bd8f7fde1784f5fa1ab24113dc27934 reported cstate change: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.808030 23135 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb01b1a8a02483e8d49cf2327886f18 with delete_type TABLET_DATA_TOMBSTONED (TS a4b53d88de2b4e938f65b317b60b0630 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41104
I20240501 20:11:29.808907 25052 tablet_replica.cc:330] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630: stopping tablet replica
I20240501 20:11:29.809324 25052 raft_consensus.cc:2229] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:29.809758 25052 raft_consensus.cc:2258] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:29.811161 25052 ts_tablet_manager.cc:1875] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:29.815611 25052 ts_tablet_manager.cc:1888] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:29.815725 25052 log.cc:1196] T 6bb01b1a8a02483e8d49cf2327886f18 P a4b53d88de2b4e938f65b317b60b0630: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/6bb01b1a8a02483e8d49cf2327886f18
I20240501 20:11:29.816447 23001 catalog_manager.cc:4830] TS a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789): tablet 6bb01b1a8a02483e8d49cf2327886f18 (table rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]) successfully deleted
I20240501 20:11:29.828508 24954 raft_consensus.cc:1050] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630: attempting to promote NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 to VOTER
I20240501 20:11:29.829211 24954 consensus_queue.cc:237] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:29.831264 23432 raft_consensus.cc:1261] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.831341 24357 raft_consensus.cc:1261] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.831564 23292 raft_consensus.cc:1261] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.831916 24954 consensus_queue.cc:1035] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.832228 25054 consensus_queue.cc:1035] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.832553 24953 consensus_queue.cc:1035] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.835258 24954 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.835784 24357 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.839049 23432 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.839239 23292 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.839731 23006 catalog_manager.cc:5484] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14 reported cstate change: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a4b53d88de2b4e938f65b317b60b0630" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.845194 23156 consensus_queue.cc:237] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:29.846854 24357 raft_consensus.cc:1261] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.847038 23432 raft_consensus.cc:1261] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.847509 24720 consensus_queue.cc:1035] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.847995 24720 consensus_queue.cc:1035] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.849820 24720 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.850584 23432 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.852439 23001 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:29.852800 24357 raft_consensus.cc:2932] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.855978 23011 catalog_manager.cc:5484] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: config changed from index 5 to 6, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "a4b53d88de2b4e938f65b317b60b0630" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:29.859536 23272 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa with delete_type TABLET_DATA_TOMBSTONED (TS 7abd52761123466ca399914a88fb23aa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38798
I20240501 20:11:29.869021 25058 tablet_replica.cc:330] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: stopping tablet replica
I20240501 20:11:29.869383 25058 raft_consensus.cc:2229] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:29.869716 25058 raft_consensus.cc:2258] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:29.871111 25058 ts_tablet_manager.cc:1875] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:29.875455 25058 ts_tablet_manager.cc:1888] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:29.875581 25058 log.cc:1196] T 0e0b80ffdd2d4d1c8f6d6c106c724efa P 7abd52761123466ca399914a88fb23aa: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/0e0b80ffdd2d4d1c8f6d6c106c724efa
I20240501 20:11:29.876361 23002 catalog_manager.cc:4830] TS 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175): tablet 0e0b80ffdd2d4d1c8f6d6c106c724efa (table rebalance_test_table_0 [id=ce9b9393a656433b9809640471389d27]) successfully deleted
I20240501 20:11:29.883939 24941 raft_consensus.cc:1050] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: attempting to promote NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 to VOTER
I20240501 20:11:29.884541 24941 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:29.886497 24225 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.886498 23292 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.886646 23156 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.886974 24941 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.887254 24942 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.887490 25048 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.891450 24942 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.892000 24225 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.892596 23156 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.896045 23014 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 reported cstate change: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.897900 23292 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.900250 23432 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:29.901809 23292 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.902041 24225 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.902369 25048 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.902678 24941 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.904513 24046 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.905081 24225 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.905164 23292 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.908123 22999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c56eb0a04104909ae46e1c7c8d732c0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:29.908826 23011 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 reported cstate change: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:29.911702 23135 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c56eb0a04104909ae46e1c7c8d732c0 with delete_type TABLET_DATA_TOMBSTONED (TS a4b53d88de2b4e938f65b317b60b0630 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41104
I20240501 20:11:29.912133 25052 tablet_replica.cc:330] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: stopping tablet replica
I20240501 20:11:29.912325 25052 raft_consensus.cc:2229] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:29.912596 25052 raft_consensus.cc:2258] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:29.913266 25052 ts_tablet_manager.cc:1875] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:29.916551 24987 raft_consensus.cc:1050] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa: attempting to promote NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c to VOTER
I20240501 20:11:29.917307 24987 consensus_queue.cc:237] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } }
I20240501 20:11:29.917932 25052 ts_tablet_manager.cc:1888] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:29.918058 25052 log.cc:1196] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/6c56eb0a04104909ae46e1c7c8d732c0
I20240501 20:11:29.918840 23001 catalog_manager.cc:4830] TS a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789): tablet 6c56eb0a04104909ae46e1c7c8d732c0 (table rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]) successfully deleted
I20240501 20:11:29.919613 23156 raft_consensus.cc:1261] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.919623 23432 raft_consensus.cc:1261] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.919742 24489 raft_consensus.cc:1261] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:29.920120 24992 consensus_queue.cc:1035] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.920367 24939 consensus_queue.cc:1035] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.920673 24940 consensus_queue.cc:1035] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:29.923285 24940 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.923938 24489 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.924455 23432 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.928050 23156 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.928172 23014 catalog_manager.cc:5484] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: config changed from index 4 to 5, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.933382 23292 consensus_queue.cc:237] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } }
I20240501 20:11:29.934756 23156 raft_consensus.cc:1261] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.934855 24489 raft_consensus.cc:1261] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:29.935271 24992 consensus_queue.cc:1035] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.935504 24939 consensus_queue.cc:1035] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:29.937465 24940 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.938060 24489 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.938100 23156 raft_consensus.cc:2932] T 6155d934dae348c99ffb489cefcd64b4 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.940964 23002 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6155d934dae348c99ffb489cefcd64b4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:29.941548 23011 catalog_manager.cc:5484] T 6155d934dae348c99ffb489cefcd64b4 P 8d0f4e53cd9e469c9d5dffee5c42ea6c reported cstate change: config changed from index 5 to 6, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } }
I20240501 20:11:29.944619 23409 tablet_service.cc:1514] Processing DeleteTablet for tablet 6155d934dae348c99ffb489cefcd64b4 with delete_type TABLET_DATA_TOMBSTONED (TS ebb07a45c9604747986a01ae6f7f7bc9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60092
I20240501 20:11:29.944939 25046 tablet_replica.cc:330] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9: stopping tablet replica
I20240501 20:11:29.945122 25046 raft_consensus.cc:2229] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:29.945459 25046 raft_consensus.cc:2258] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:29.946236 25046 ts_tablet_manager.cc:1875] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:29.950160 25046 ts_tablet_manager.cc:1888] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:29.950253 25046 log.cc:1196] T 6155d934dae348c99ffb489cefcd64b4 P ebb07a45c9604747986a01ae6f7f7bc9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/6155d934dae348c99ffb489cefcd64b4
I20240501 20:11:29.950888 22999 catalog_manager.cc:4830] TS ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603): tablet 6155d934dae348c99ffb489cefcd64b4 (table rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]) successfully deleted
I20240501 20:11:29.985483 23292 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df"
dest_uuid: "7abd52761123466ca399914a88fb23aa"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35344
I20240501 20:11:29.985656 23292 raft_consensus.cc:595] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:30.031663 24939 raft_consensus.cc:979] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: : Instructing follower 527d2c1d7b3d45aeb992646b89911c14 to start an election
I20240501 20:11:30.031844 24939 raft_consensus.cc:1067] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Signalling peer 527d2c1d7b3d45aeb992646b89911c14 to start an election
I20240501 20:11:30.032680 24357 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df"
dest_uuid: "527d2c1d7b3d45aeb992646b89911c14"
 from {username='slave'} at 127.8.68.194:34443
I20240501 20:11:30.032940 24357 raft_consensus.cc:482] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:30.033138 24357 raft_consensus.cc:3037] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:30.035560 24357 raft_consensus.cc:504] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:30.037835 23292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df" candidate_uuid: "527d2c1d7b3d45aeb992646b89911c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7abd52761123466ca399914a88fb23aa"
I20240501 20:11:30.038053 23292 raft_consensus.cc:3032] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:30.037982 23156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df" candidate_uuid: "527d2c1d7b3d45aeb992646b89911c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:30.038138 23292 raft_consensus.cc:729] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Running, Role: LEADER
I20240501 20:11:30.038242 23156 raft_consensus.cc:3037] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:30.038498 23292 consensus_queue.cc:260] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:30.038956 23292 raft_consensus.cc:3037] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:30.040228 24357 leader_election.cc:288] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [CANDIDATE]: Term 2 election: Requested vote from peers 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175), a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:30.040413 23156 raft_consensus.cc:2454] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d2c1d7b3d45aeb992646b89911c14 in term 2.
I20240501 20:11:30.041785 23292 raft_consensus.cc:2454] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d2c1d7b3d45aeb992646b89911c14 in term 2.
I20240501 20:11:30.042464 24293 leader_election.cc:302] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 527d2c1d7b3d45aeb992646b89911c14, 7abd52761123466ca399914a88fb23aa, a4b53d88de2b4e938f65b317b60b0630; no voters: 
I20240501 20:11:30.042907 24833 raft_consensus.cc:2781] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:30.043417 24833 raft_consensus.cc:686] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Becoming Leader. State: Replica: 527d2c1d7b3d45aeb992646b89911c14, State: Running, Role: LEADER
I20240501 20:11:30.044020 24833 consensus_queue.cc:237] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:30.052026 23013 catalog_manager.cc:5484] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 reported cstate change: term changed from 1 to 2, leader changed from 7abd52761123466ca399914a88fb23aa (127.8.68.194) to 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198). New cstate: current_term: 2 leader_uuid: "527d2c1d7b3d45aeb992646b89911c14" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:30.057528 23432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940d4d4f1a54388b49f6794ca57d0df" candidate_uuid: "527d2c1d7b3d45aeb992646b89911c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
I20240501 20:11:30.057770 23432 raft_consensus.cc:3037] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:30.060132 23432 raft_consensus.cc:2454] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d2c1d7b3d45aeb992646b89911c14 in term 2.
I20240501 20:11:30.115388 24940 raft_consensus.cc:1050] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: attempting to promote NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 to VOTER
I20240501 20:11:30.116041 24940 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:30.118117 24225 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 LEARNER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:30.118165 23432 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:30.118156 23156 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:30.118670 24687 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:30.118963 24939 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:30.119253 24992 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:30.122004 24687 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:30.122604 23432 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:30.125206 24225 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:30.125483 23156 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:30.127318 23013 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index 4 to 5, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:30.130596 24687 raft_consensus.cc:1050] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: attempting to promote NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 to VOTER
I20240501 20:11:30.131531 24687 consensus_queue.cc:237] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:30.132349 23292 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:30.133867 23156 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:30.134773 23432 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:30.135076 24225 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:30.135319 24987 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:30.135361 23431 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:30.135792 24987 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:30.136127 24357 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:30.136214 24987 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:30.136435 24992 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:30.137951 24940 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:30.138513 24992 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:30.139616 23430 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:30.142199 23002 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7475f202d7f2476fbd69ef838d7e1ee2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:30.143771 24687 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:30.144033 23014 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:30.145263 24225 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:30.147410 23135 tablet_service.cc:1514] Processing DeleteTablet for tablet 7475f202d7f2476fbd69ef838d7e1ee2 with delete_type TABLET_DATA_TOMBSTONED (TS a4b53d88de2b4e938f65b317b60b0630 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41104
I20240501 20:11:30.147764 25052 tablet_replica.cc:330] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: stopping tablet replica
I20240501 20:11:30.147958 25052 raft_consensus.cc:2229] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:30.148276 25052 raft_consensus.cc:2258] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:30.148310 23430 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:30.148674 24357 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:30.150333 23014 catalog_manager.cc:5484] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:30.151058 25052 ts_tablet_manager.cc:1875] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:30.154901 23156 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:30.157821 25052 ts_tablet_manager.cc:1888] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:30.157943 25052 log.cc:1196] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/7475f202d7f2476fbd69ef838d7e1ee2
I20240501 20:11:30.158669 23001 catalog_manager.cc:4830] TS a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789): tablet 7475f202d7f2476fbd69ef838d7e1ee2 (table rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]) successfully deleted
I20240501 20:11:30.300029 23292 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7"
dest_uuid: "7abd52761123466ca399914a88fb23aa"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35444
I20240501 20:11:30.300204 23292 raft_consensus.cc:595] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:30.327765 23292 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7"
dest_uuid: "7abd52761123466ca399914a88fb23aa"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35476
I20240501 20:11:30.327930 23292 raft_consensus.cc:595] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:30.452333 23292 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:30.452932 24833 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:30.485834 23156 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:30.486449 24833 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:30.492213 23430 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:30.494968 25179 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:30.551256 23292 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7"
dest_uuid: "7abd52761123466ca399914a88fb23aa"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35490
I20240501 20:11:30.551448 23292 raft_consensus.cc:595] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:30.564736 24992 raft_consensus.cc:979] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: : Instructing follower 527d2c1d7b3d45aeb992646b89911c14 to start an election
I20240501 20:11:30.564888 24992 raft_consensus.cc:1067] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Signalling peer 527d2c1d7b3d45aeb992646b89911c14 to start an election
I20240501 20:11:30.565472 24357 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7"
dest_uuid: "527d2c1d7b3d45aeb992646b89911c14"
 from {username='slave'} at 127.8.68.194:34443
I20240501 20:11:30.565654 24357 raft_consensus.cc:482] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:30.565735 24357 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:30.567544 24357 raft_consensus.cc:504] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:30.568192 24357 leader_election.cc:288] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [CANDIDATE]: Term 2 election: Requested vote from peers 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175), a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789), ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:30.568544 23292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "527d2c1d7b3d45aeb992646b89911c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7abd52761123466ca399914a88fb23aa"
I20240501 20:11:30.568624 23156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "527d2c1d7b3d45aeb992646b89911c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
I20240501 20:11:30.568756 23292 raft_consensus.cc:3032] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:30.568830 23292 raft_consensus.cc:729] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Running, Role: LEADER
I20240501 20:11:30.568838 23156 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:30.568800 23430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "527d2c1d7b3d45aeb992646b89911c14" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
I20240501 20:11:30.569028 23430 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:30.569096 23292 consensus_queue.cc:260] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:30.569533 23292 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:30.570921 23430 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d2c1d7b3d45aeb992646b89911c14 in term 2.
I20240501 20:11:30.571449 23156 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d2c1d7b3d45aeb992646b89911c14 in term 2.
I20240501 20:11:30.571870 24292 leader_election.cc:302] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 527d2c1d7b3d45aeb992646b89911c14, a4b53d88de2b4e938f65b317b60b0630, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:30.572132 23292 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d2c1d7b3d45aeb992646b89911c14 in term 2.
I20240501 20:11:30.572185 25118 raft_consensus.cc:2781] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:30.572335 25118 raft_consensus.cc:686] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Becoming Leader. State: Replica: 527d2c1d7b3d45aeb992646b89911c14, State: Running, Role: LEADER
I20240501 20:11:30.572620 25118 consensus_queue.cc:237] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:30.574956 23011 catalog_manager.cc:5484] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 reported cstate change: term changed from 1 to 2, leader changed from 7abd52761123466ca399914a88fb23aa (127.8.68.194) to 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198). New cstate: current_term: 2 leader_uuid: "527d2c1d7b3d45aeb992646b89911c14" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:30.959411 24357 consensus_queue.cc:237] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:30.961284 23430 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:30.961285 23156 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:30.961864 25118 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:30.962198 25181 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:30.964226 25179 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:30.964915 23430 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:30.964844 23156 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:30.968781 22999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d940d4d4f1a54388b49f6794ca57d0df with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:30.969547 23014 catalog_manager.cc:5484] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index 5 to 7, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "527d2c1d7b3d45aeb992646b89911c14" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:30.973080 23272 tablet_service.cc:1514] Processing DeleteTablet for tablet d940d4d4f1a54388b49f6794ca57d0df with delete_type TABLET_DATA_TOMBSTONED (TS 7abd52761123466ca399914a88fb23aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38798
I20240501 20:11:30.978647 25213 tablet_replica.cc:330] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: stopping tablet replica
I20240501 20:11:30.978827 25213 raft_consensus.cc:2229] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:30.979044 25213 raft_consensus.cc:2258] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:30.979889 25213 ts_tablet_manager.cc:1875] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:30.985570 25213 ts_tablet_manager.cc:1888] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:11:30.985702 25213 log.cc:1196] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/d940d4d4f1a54388b49f6794ca57d0df
I20240501 20:11:30.986507 23002 catalog_manager.cc:4830] TS 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175): tablet d940d4d4f1a54388b49f6794ca57d0df (table rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]) successfully deleted
I20240501 20:11:30.996868 23292 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:30.997452 25179 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:31.024729 23430 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:31.025238 25179 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:31.028286 23156 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:31.028874 25118 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:31.590101 24357 consensus_queue.cc:237] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } }
I20240501 20:11:31.591704 23156 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:31.591730 23430 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:31.592200 25239 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:31.592469 25240 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:31.594475 25239 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:31.595216 23156 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:31.595391 23430 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:31.598273 22999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d3d2792048e4b4aa59c034e0539ddb7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:31.599541 23014 catalog_manager.cc:5484] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index 5 to 7, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New cstate: current_term: 2 leader_uuid: "527d2c1d7b3d45aeb992646b89911c14" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } }
I20240501 20:11:31.605610 23272 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3d2792048e4b4aa59c034e0539ddb7 with delete_type TABLET_DATA_TOMBSTONED (TS 7abd52761123466ca399914a88fb23aa not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38798
I20240501 20:11:31.606592 25245 tablet_replica.cc:330] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: stopping tablet replica
I20240501 20:11:31.606765 25245 raft_consensus.cc:2229] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:31.606938 25245 raft_consensus.cc:2258] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:31.607671 25245 ts_tablet_manager.cc:1875] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:31.611721 25245 ts_tablet_manager.cc:1888] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:11:31.611824 25245 log.cc:1196] T 3d3d2792048e4b4aa59c034e0539ddb7 P 7abd52761123466ca399914a88fb23aa: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/3d3d2792048e4b4aa59c034e0539ddb7
I20240501 20:11:31.612494 23002 catalog_manager.cc:4830] TS 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175): tablet 3d3d2792048e4b4aa59c034e0539ddb7 (table rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]) successfully deleted
I20240501 20:11:31.718238 24338 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:31.733843 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:31.745867 23272 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:31.773089 23135 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:31.773510 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:31.787961 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:31.930199 24338 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:31.943300 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:31.943473 23272 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:31.945235 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:31.958861 23135 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:31.973172 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:32.039336 23292 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } }
W20240501 20:11:32.042263 23224 consensus_peers.cc:485] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa -> Peer 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197:33911): Couldn't send request to peer 8d0f4e53cd9e469c9d5dffee5c42ea6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7475f202d7f2476fbd69ef838d7e1ee2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.043062 24225 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:32.043805 24992 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:32.053092 23430 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:32.053668 25338 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:32.058265 24992 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:32.058933 24225 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:32.059706 23430 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:32.063688 23011 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: config changed from index 6 to 7, NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } }
I20240501 20:11:32.087329 24357 consensus_queue.cc:237] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:32.089768 23156 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:11:32.090052 23430 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:11:32.090256 25118 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:32.090688 25241 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:32.092619 25118 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:32.093626 23156 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:32.095985 23430 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } } }
I20240501 20:11:32.099292 23013 catalog_manager.cc:5484] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 reported cstate change: config changed from index 7 to 8, NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) added. New cstate: current_term: 2 leader_uuid: "527d2c1d7b3d45aeb992646b89911c14" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:32.102694 24290 consensus_peers.cc:485] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 -> Peer 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196:33701): Couldn't send request to peer 5bd8f7fde1784f5fa1ab24113dc27934. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d3d2792048e4b4aa59c034e0539ddb7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.119144 23430 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } }
I20240501 20:11:32.122462 24225 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:32.123124 23292 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:32.123185 24941 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:32.123911 24941 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:32.126660 24941 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:32.127058 23292 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
W20240501 20:11:32.129346 23358 consensus_peers.cc:485] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 -> Peer 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689): Couldn't send request to peer 527d2c1d7b3d45aeb992646b89911c14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c56eb0a04104909ae46e1c7c8d732c0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.132333 23011 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index 6 to 7, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:32.143373 24225 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } }
I20240501 20:11:32.196349 24338 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:32.203032 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:32.209992 23272 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:32.239261 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:32.241899 23135 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:32.253482 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:32.365795 23292 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.368196 24225 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:11:32.368552 23226 consensus_peers.cc:485] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa -> Peer a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789): Couldn't send request to peer a4b53d88de2b4e938f65b317b60b0630. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.368727 23430 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:32.368942 24992 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:11:32.369325 23224 consensus_peers.cc:485] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa -> Peer 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197:33911): Couldn't send request to peer 8d0f4e53cd9e469c9d5dffee5c42ea6c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7475f202d7f2476fbd69ef838d7e1ee2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.369681 25408 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:32.371587 24992 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:32.372321 24225 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:32.375197 23430 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:32.379369 23014 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index 7 to 8, NON_VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) added. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:32.388670 23430 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.390648 23292 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:32.390892 24225 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:11:32.390909 23360 consensus_peers.cc:485] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 -> Peer a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789): Couldn't send request to peer a4b53d88de2b4e938f65b317b60b0630. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.391161 25371 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:11:32.391425 23358 consensus_peers.cc:485] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 -> Peer 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689): Couldn't send request to peer 527d2c1d7b3d45aeb992646b89911c14. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c56eb0a04104909ae46e1c7c8d732c0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.391669 25371 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:32.393132 24941 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:32.393653 23292 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:32.396919 24225 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:32.397048 23011 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: config changed from index 7 to 8, NON_VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) added. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:32.414069 24357 consensus_queue.cc:237] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: true } }
I20240501 20:11:32.416083 23430 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:11:32.416414 23156 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:11:32.416594 25118 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:11:32.416904 24293 consensus_peers.cc:485] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 -> Peer 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175): Couldn't send request to peer 7abd52761123466ca399914a88fb23aa. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.416970 25241 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:32.419085 25118 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: true } } }
I20240501 20:11:32.420040 23156 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: true } } }
I20240501 20:11:32.422562 23430 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: true } } }
I20240501 20:11:32.425000 23014 catalog_manager.cc:5484] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index 7 to 8, NON_VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) added. New cstate: current_term: 2 leader_uuid: "527d2c1d7b3d45aeb992646b89911c14" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: true } } }
I20240501 20:11:32.630992 25452 ts_tablet_manager.cc:902] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: Initiating tablet copy from peer 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689)
I20240501 20:11:32.632191 25452 tablet_copy_client.cc:308] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.198:43689
I20240501 20:11:32.637796 24378 tablet_copy_service.cc:140] P 527d2c1d7b3d45aeb992646b89911c14: Received BeginTabletCopySession request for tablet 3d3d2792048e4b4aa59c034e0539ddb7 from peer 5bd8f7fde1784f5fa1ab24113dc27934 ({username='slave'} at 127.8.68.196:32849)
I20240501 20:11:32.637965 24378 tablet_copy_service.cc:161] P 527d2c1d7b3d45aeb992646b89911c14: Beginning new tablet copy session on tablet 3d3d2792048e4b4aa59c034e0539ddb7 from peer 5bd8f7fde1784f5fa1ab24113dc27934 at {username='slave'} at 127.8.68.196:32849: session id = 5bd8f7fde1784f5fa1ab24113dc27934-3d3d2792048e4b4aa59c034e0539ddb7
I20240501 20:11:32.640524 24378 tablet_copy_source_session.cc:205] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:32.641891 25452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3d2792048e4b4aa59c034e0539ddb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.646224 25452 tablet_copy_client.cc:783] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:32.646461 25452 tablet_copy_client.cc:647] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:32.648084 25452 tablet_copy_client.cc:523] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:32.650168 25452 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap starting.
I20240501 20:11:32.693082 25452 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:32.693500 25452 tablet_bootstrap.cc:492] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: Bootstrap complete.
I20240501 20:11:32.693779 25452 ts_tablet_manager.cc:1367] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: Time spent bootstrapping tablet: real 0.044s	user 0.041s	sys 0.000s
I20240501 20:11:32.694437 25452 raft_consensus.cc:348] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:32.694629 25452 raft_consensus.cc:729] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5bd8f7fde1784f5fa1ab24113dc27934, State: Initialized, Role: LEARNER
I20240501 20:11:32.694777 25452 consensus_queue.cc:260] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: true } }
I20240501 20:11:32.695223 25452 ts_tablet_manager.cc:1398] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240501 20:11:32.695967 24378 tablet_copy_service.cc:342] P 527d2c1d7b3d45aeb992646b89911c14: Request end of tablet copy session 5bd8f7fde1784f5fa1ab24113dc27934-3d3d2792048e4b4aa59c034e0539ddb7 received from {username='slave'} at 127.8.68.196:32849
I20240501 20:11:32.696105 24378 tablet_copy_service.cc:433] P 527d2c1d7b3d45aeb992646b89911c14: ending tablet copy session 5bd8f7fde1784f5fa1ab24113dc27934-3d3d2792048e4b4aa59c034e0539ddb7 on tablet 3d3d2792048e4b4aa59c034e0539ddb7 with peer 5bd8f7fde1784f5fa1ab24113dc27934
I20240501 20:11:32.784229 25489 ts_tablet_manager.cc:902] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Initiating tablet copy from peer 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:32.785657 25489 tablet_copy_client.cc:308] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:41175
I20240501 20:11:32.786401 23313 tablet_copy_service.cc:140] P 7abd52761123466ca399914a88fb23aa: Received BeginTabletCopySession request for tablet 7475f202d7f2476fbd69ef838d7e1ee2 from peer a4b53d88de2b4e938f65b317b60b0630 ({username='slave'} at 127.8.68.193:60757)
I20240501 20:11:32.786540 23313 tablet_copy_service.cc:161] P 7abd52761123466ca399914a88fb23aa: Beginning new tablet copy session on tablet 7475f202d7f2476fbd69ef838d7e1ee2 from peer a4b53d88de2b4e938f65b317b60b0630 at {username='slave'} at 127.8.68.193:60757: session id = a4b53d88de2b4e938f65b317b60b0630-7475f202d7f2476fbd69ef838d7e1ee2
I20240501 20:11:32.788479 23313 tablet_copy_source_session.cc:205] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:32.789646 25489 ts_tablet_manager.cc:1875] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:11:32.794102 25489 ts_tablet_manager.cc:1888] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:11:32.794373 25489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7475f202d7f2476fbd69ef838d7e1ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.796495 25489 tablet_copy_client.cc:783] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:32.796722 25489 tablet_copy_client.cc:647] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:32.798435 25489 tablet_copy_client.cc:523] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:32.800889 25489 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:32.833128 25489 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:32.833467 25489 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap complete.
I20240501 20:11:32.833681 25489 ts_tablet_manager.cc:1367] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.033s	user 0.023s	sys 0.009s
I20240501 20:11:32.834260 25489 raft_consensus.cc:348] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.834523 25489 raft_consensus.cc:729] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: LEARNER
I20240501 20:11:32.834709 25489 consensus_queue.cc:260] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.835219 25489 ts_tablet_manager.cc:1398] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.003s
I20240501 20:11:32.836961 23313 tablet_copy_service.cc:342] P 7abd52761123466ca399914a88fb23aa: Request end of tablet copy session a4b53d88de2b4e938f65b317b60b0630-7475f202d7f2476fbd69ef838d7e1ee2 received from {username='slave'} at 127.8.68.193:60757
I20240501 20:11:32.837092 23313 tablet_copy_service.cc:433] P 7abd52761123466ca399914a88fb23aa: ending tablet copy session a4b53d88de2b4e938f65b317b60b0630-7475f202d7f2476fbd69ef838d7e1ee2 on tablet 7475f202d7f2476fbd69ef838d7e1ee2 with peer a4b53d88de2b4e938f65b317b60b0630
I20240501 20:11:32.844424 25495 ts_tablet_manager.cc:902] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: Initiating tablet copy from peer ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:32.845695 25495 tablet_copy_client.cc:308] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:45603
I20240501 20:11:32.846374 23451 tablet_copy_service.cc:140] P ebb07a45c9604747986a01ae6f7f7bc9: Received BeginTabletCopySession request for tablet 6c56eb0a04104909ae46e1c7c8d732c0 from peer 527d2c1d7b3d45aeb992646b89911c14 ({username='slave'} at 127.8.68.198:41169)
I20240501 20:11:32.846509 23451 tablet_copy_service.cc:161] P ebb07a45c9604747986a01ae6f7f7bc9: Beginning new tablet copy session on tablet 6c56eb0a04104909ae46e1c7c8d732c0 from peer 527d2c1d7b3d45aeb992646b89911c14 at {username='slave'} at 127.8.68.198:41169: session id = 527d2c1d7b3d45aeb992646b89911c14-6c56eb0a04104909ae46e1c7c8d732c0
I20240501 20:11:32.848492 23451 tablet_copy_source_session.cc:205] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:32.849655 25495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c56eb0a04104909ae46e1c7c8d732c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.855835 25495 tablet_copy_client.cc:783] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:32.856060 25495 tablet_copy_client.cc:647] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:32.857712 25495 tablet_copy_client.cc:523] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:32.859959 25495 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap starting.
I20240501 20:11:32.891144 25495 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:32.891482 25495 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: Bootstrap complete.
I20240501 20:11:32.891700 25495 ts_tablet_manager.cc:1367] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: Time spent bootstrapping tablet: real 0.032s	user 0.031s	sys 0.001s
I20240501 20:11:32.892268 25495 raft_consensus.cc:348] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.892549 25495 raft_consensus.cc:729] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 527d2c1d7b3d45aeb992646b89911c14, State: Initialized, Role: LEARNER
I20240501 20:11:32.892746 25495 consensus_queue.cc:260] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.893721 25495 ts_tablet_manager.cc:1398] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:32.894470 25489 ts_tablet_manager.cc:902] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Initiating tablet copy from peer ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603)
I20240501 20:11:32.894556 23451 tablet_copy_service.cc:342] P ebb07a45c9604747986a01ae6f7f7bc9: Request end of tablet copy session 527d2c1d7b3d45aeb992646b89911c14-6c56eb0a04104909ae46e1c7c8d732c0 received from {username='slave'} at 127.8.68.198:41169
I20240501 20:11:32.894693 23451 tablet_copy_service.cc:433] P ebb07a45c9604747986a01ae6f7f7bc9: ending tablet copy session 527d2c1d7b3d45aeb992646b89911c14-6c56eb0a04104909ae46e1c7c8d732c0 on tablet 6c56eb0a04104909ae46e1c7c8d732c0 with peer 527d2c1d7b3d45aeb992646b89911c14
I20240501 20:11:32.896956 25489 tablet_copy_client.cc:308] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:45603
I20240501 20:11:32.897560 23451 tablet_copy_service.cc:140] P ebb07a45c9604747986a01ae6f7f7bc9: Received BeginTabletCopySession request for tablet 6c56eb0a04104909ae46e1c7c8d732c0 from peer a4b53d88de2b4e938f65b317b60b0630 ({username='slave'} at 127.8.68.193:53649)
I20240501 20:11:32.897730 23451 tablet_copy_service.cc:161] P ebb07a45c9604747986a01ae6f7f7bc9: Beginning new tablet copy session on tablet 6c56eb0a04104909ae46e1c7c8d732c0 from peer a4b53d88de2b4e938f65b317b60b0630 at {username='slave'} at 127.8.68.193:53649: session id = a4b53d88de2b4e938f65b317b60b0630-6c56eb0a04104909ae46e1c7c8d732c0
I20240501 20:11:32.899958 23451 tablet_copy_source_session.cc:205] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:32.900928 25489 ts_tablet_manager.cc:1875] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:11:32.907470 25489 ts_tablet_manager.cc:1888] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:11:32.907732 25489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c56eb0a04104909ae46e1c7c8d732c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.910151 25489 tablet_copy_client.cc:783] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:32.910426 25489 tablet_copy_client.cc:647] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:32.912101 25489 tablet_copy_client.cc:523] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:32.915146 25489 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap starting.
I20240501 20:11:32.939029 25521 ts_tablet_manager.cc:902] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Initiating tablet copy from peer 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175)
I20240501 20:11:32.940325 25521 tablet_copy_client.cc:308] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:41175
I20240501 20:11:32.940932 23313 tablet_copy_service.cc:140] P 7abd52761123466ca399914a88fb23aa: Received BeginTabletCopySession request for tablet 7475f202d7f2476fbd69ef838d7e1ee2 from peer 8d0f4e53cd9e469c9d5dffee5c42ea6c ({username='slave'} at 127.8.68.197:60865)
I20240501 20:11:32.941057 23313 tablet_copy_service.cc:161] P 7abd52761123466ca399914a88fb23aa: Beginning new tablet copy session on tablet 7475f202d7f2476fbd69ef838d7e1ee2 from peer 8d0f4e53cd9e469c9d5dffee5c42ea6c at {username='slave'} at 127.8.68.197:60865: session id = 8d0f4e53cd9e469c9d5dffee5c42ea6c-7475f202d7f2476fbd69ef838d7e1ee2
I20240501 20:11:32.942965 23313 tablet_copy_source_session.cc:205] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:32.944034 25521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7475f202d7f2476fbd69ef838d7e1ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:32.947597 25489 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:32.947937 25489 tablet_bootstrap.cc:492] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Bootstrap complete.
I20240501 20:11:32.948138 25489 ts_tablet_manager.cc:1367] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Time spent bootstrapping tablet: real 0.033s	user 0.032s	sys 0.000s
I20240501 20:11:32.948381 25521 tablet_copy_client.cc:783] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:32.948624 25521 tablet_copy_client.cc:647] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:32.948799 25489 raft_consensus.cc:348] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.949007 25489 raft_consensus.cc:729] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Initialized, Role: LEARNER
I20240501 20:11:32.949254 25489 consensus_queue.cc:260] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: NON_VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.949751 25489 ts_tablet_manager.cc:1398] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:32.950125 25521 tablet_copy_client.cc:523] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:32.950541 23451 tablet_copy_service.cc:342] P ebb07a45c9604747986a01ae6f7f7bc9: Request end of tablet copy session a4b53d88de2b4e938f65b317b60b0630-6c56eb0a04104909ae46e1c7c8d732c0 received from {username='slave'} at 127.8.68.193:53649
I20240501 20:11:32.950654 23451 tablet_copy_service.cc:433] P ebb07a45c9604747986a01ae6f7f7bc9: ending tablet copy session a4b53d88de2b4e938f65b317b60b0630-6c56eb0a04104909ae46e1c7c8d732c0 on tablet 6c56eb0a04104909ae46e1c7c8d732c0 with peer a4b53d88de2b4e938f65b317b60b0630
I20240501 20:11:32.953403 25521 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Bootstrap starting.
I20240501 20:11:32.984366 25521 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:32.984799 25521 tablet_bootstrap.cc:492] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Bootstrap complete.
I20240501 20:11:32.985018 25521 ts_tablet_manager.cc:1367] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Time spent bootstrapping tablet: real 0.032s	user 0.027s	sys 0.006s
I20240501 20:11:32.985558 25521 raft_consensus.cc:348] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.985766 25521 raft_consensus.cc:729] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d0f4e53cd9e469c9d5dffee5c42ea6c, State: Initialized, Role: LEARNER
I20240501 20:11:32.985956 25521 consensus_queue.cc:260] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:32.986691 25521 ts_tablet_manager.cc:1398] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 20:11:32.987489 23313 tablet_copy_service.cc:342] P 7abd52761123466ca399914a88fb23aa: Request end of tablet copy session 8d0f4e53cd9e469c9d5dffee5c42ea6c-7475f202d7f2476fbd69ef838d7e1ee2 received from {username='slave'} at 127.8.68.197:60865
I20240501 20:11:32.987643 23313 tablet_copy_service.cc:433] P 7abd52761123466ca399914a88fb23aa: ending tablet copy session 8d0f4e53cd9e469c9d5dffee5c42ea6c-7475f202d7f2476fbd69ef838d7e1ee2 on tablet 7475f202d7f2476fbd69ef838d7e1ee2 with peer 8d0f4e53cd9e469c9d5dffee5c42ea6c
I20240501 20:11:33.004680 25538 ts_tablet_manager.cc:902] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Initiating tablet copy from peer 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689)
I20240501 20:11:33.006593 25538 tablet_copy_client.cc:308] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.198:43689
I20240501 20:11:33.007321 24378 tablet_copy_service.cc:140] P 527d2c1d7b3d45aeb992646b89911c14: Received BeginTabletCopySession request for tablet d940d4d4f1a54388b49f6794ca57d0df from peer 7abd52761123466ca399914a88fb23aa ({username='slave'} at 127.8.68.194:34443)
I20240501 20:11:33.007481 24378 tablet_copy_service.cc:161] P 527d2c1d7b3d45aeb992646b89911c14: Beginning new tablet copy session on tablet d940d4d4f1a54388b49f6794ca57d0df from peer 7abd52761123466ca399914a88fb23aa at {username='slave'} at 127.8.68.194:34443: session id = 7abd52761123466ca399914a88fb23aa-d940d4d4f1a54388b49f6794ca57d0df
I20240501 20:11:33.009505 24378 tablet_copy_source_session.cc:205] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:33.010861 25538 ts_tablet_manager.cc:1875] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:11:33.015771 25538 ts_tablet_manager.cc:1888] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 20:11:33.016032 25538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940d4d4f1a54388b49f6794ca57d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:33.018122 25538 tablet_copy_client.cc:783] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:33.018457 25538 tablet_copy_client.cc:647] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:33.020200 25538 tablet_copy_client.cc:523] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:33.023046 25538 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Bootstrap starting.
I20240501 20:11:33.054481 25538 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:33.054834 25538 tablet_bootstrap.cc:492] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Bootstrap complete.
I20240501 20:11:33.055058 25538 ts_tablet_manager.cc:1367] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Time spent bootstrapping tablet: real 0.032s	user 0.028s	sys 0.004s
I20240501 20:11:33.055653 25538 raft_consensus.cc:348] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: true } }
I20240501 20:11:33.055856 25538 raft_consensus.cc:729] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7abd52761123466ca399914a88fb23aa, State: Initialized, Role: LEARNER
I20240501 20:11:33.056039 25538 consensus_queue.cc:260] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: NON_VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: true } }
I20240501 20:11:33.056499 25538 ts_tablet_manager.cc:1398] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:33.057286 24378 tablet_copy_service.cc:342] P 527d2c1d7b3d45aeb992646b89911c14: Request end of tablet copy session 7abd52761123466ca399914a88fb23aa-d940d4d4f1a54388b49f6794ca57d0df received from {username='slave'} at 127.8.68.194:34443
I20240501 20:11:33.057389 24378 tablet_copy_service.cc:433] P 527d2c1d7b3d45aeb992646b89911c14: ending tablet copy session 7abd52761123466ca399914a88fb23aa-d940d4d4f1a54388b49f6794ca57d0df on tablet d940d4d4f1a54388b49f6794ca57d0df with peer 7abd52761123466ca399914a88fb23aa
I20240501 20:11:33.077361 24225 raft_consensus.cc:1203] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240501 20:11:33.325152 23156 raft_consensus.cc:1203] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:33.366101 24357 raft_consensus.cc:1203] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:33.453096 23156 raft_consensus.cc:1203] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:33.456544 23293 raft_consensus.cc:1203] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240501 20:11:33.467123 25549 raft_consensus.cc:1050] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: attempting to promote NON_VOTER 5bd8f7fde1784f5fa1ab24113dc27934 to VOTER
I20240501 20:11:33.467890 25549 consensus_queue.cc:237] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:33.470037 23156 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:33.470171 23430 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:33.470556 24225 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 2 LEARNER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:33.470682 25611 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.471001 25118 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.471518 25118 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.476015 24489 raft_consensus.cc:1203] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:33.482118 25118 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:33.482718 24225 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:33.485481 23156 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:33.486619 23014 catalog_manager.cc:5484] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 reported cstate change: config changed from index 8 to 9, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "527d2c1d7b3d45aeb992646b89911c14" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:33.486930 23430 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:33.507629 24357 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7"
dest_uuid: "527d2c1d7b3d45aeb992646b89911c14"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:44696
I20240501 20:11:33.507835 24357 raft_consensus.cc:595] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Received request to transfer leadership
I20240501 20:11:33.739773 24357 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7"
dest_uuid: "527d2c1d7b3d45aeb992646b89911c14"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:44730
I20240501 20:11:33.740018 24357 raft_consensus.cc:595] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Received request to transfer leadership
I20240501 20:11:33.822773 25594 raft_consensus.cc:1050] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: attempting to promote NON_VOTER 527d2c1d7b3d45aeb992646b89911c14 to VOTER
I20240501 20:11:33.823493 25594 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }
I20240501 20:11:33.825804 24225 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:33.825790 23156 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:33.826339 24357 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 LEARNER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:33.826458 23293 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:33.826465 25680 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.826881 25363 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.827150 25593 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.827399 25680 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.830365 25593 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:33.830792 24225 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:33.831494 23156 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:33.832119 24357 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:33.834549 25593 raft_consensus.cc:1050] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9: attempting to promote NON_VOTER a4b53d88de2b4e938f65b317b60b0630 to VOTER
I20240501 20:11:33.834666 23293 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:33.835186 25593 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } }
I20240501 20:11:33.835882 23013 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 reported cstate change: config changed from index 8 to 9, 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } } }
I20240501 20:11:33.838027 25408 raft_consensus.cc:1050] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: attempting to promote NON_VOTER a4b53d88de2b4e938f65b317b60b0630 to VOTER
I20240501 20:11:33.838976 25408 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } }
I20240501 20:11:33.840085 24225 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:33.842439 24489 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:33.842856 23155 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:33.842878 24225 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:33.842993 25408 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.843302 23430 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:33.843712 25594 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:33.843757 25581 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.844175 25581 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.844183 25594 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:11:33.845038 22999 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 6c56eb0a04104909ae46e1c7c8d732c0 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: NON_VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: true } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } }
I20240501 20:11:33.850564 23293 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:33.851156 23156 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 LEARNER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:33.852177 25408 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.852790 25593 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:33.852794 24224 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.853772 25581 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.854924 25594 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.855527 23293 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.858996 23155 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.859488 24225 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.860244 23430 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.863243 23006 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: config changed from index 9 to 10, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:33.864763 23011 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 reported cstate change: config changed from index 8 to 9, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.866091 24358 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:33.867075 25594 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:33.868631 24358 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.871256 23156 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.883185 23430 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } }
I20240501 20:11:33.883940 25581 raft_consensus.cc:1050] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa: attempting to promote NON_VOTER 8d0f4e53cd9e469c9d5dffee5c42ea6c to VOTER
I20240501 20:11:33.884801 25581 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } }
I20240501 20:11:33.885366 24357 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:11:33.885455 24225 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:11:33.885970 23156 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:11:33.886309 25680 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:11:33.886786 25680 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:11:33.887135 25594 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:11:33.888337 24225 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:33.889169 25582 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:33.889422 24488 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:33.889804 24489 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: true } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.890730 25582 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:33.891747 23155 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:33.892239 25582 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:33.893679 23430 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:33.893847 25593 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.894256 25581 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:33.895769 25582 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.898013 23430 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.898674 23157 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.899482 24224 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.899852 23155 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.901311 22999 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 6c56eb0a04104909ae46e1c7c8d732c0 with cas_config_opid_index 9: aborting the task: latest config opid_index 10; task opid_index 9
I20240501 20:11:33.901602 22999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c56eb0a04104909ae46e1c7c8d732c0 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:33.904314 25611 raft_consensus.cc:979] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: : Instructing follower a4b53d88de2b4e938f65b317b60b0630 to start an election
I20240501 20:11:33.904470 25611 raft_consensus.cc:1067] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Signalling peer a4b53d88de2b4e938f65b317b60b0630 to start an election
I20240501 20:11:33.905254 24225 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.905298 23013 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index 10 to 11, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.907435 23156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7"
dest_uuid: "a4b53d88de2b4e938f65b317b60b0630"
 from {username='slave'} at 127.8.68.198:34399
I20240501 20:11:33.907655 23156 raft_consensus.cc:482] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:33.907773 23156 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:33.910002 23272 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c56eb0a04104909ae46e1c7c8d732c0 with delete_type TABLET_DATA_TOMBSTONED (TS 7abd52761123466ca399914a88fb23aa not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:38798
I20240501 20:11:33.910934 23156 raft_consensus.cc:504] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:33.911048 24488 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.912621 23430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "a4b53d88de2b4e938f65b317b60b0630" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "ebb07a45c9604747986a01ae6f7f7bc9"
I20240501 20:11:33.912794 23430 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:33.914959 24358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "a4b53d88de2b4e938f65b317b60b0630" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "527d2c1d7b3d45aeb992646b89911c14"
I20240501 20:11:33.915273 24358 raft_consensus.cc:3032] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:33.915275 23430 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b53d88de2b4e938f65b317b60b0630 in term 3.
I20240501 20:11:33.915359 24358 raft_consensus.cc:729] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 527d2c1d7b3d45aeb992646b89911c14, State: Running, Role: LEADER
I20240501 20:11:33.915715 24358 consensus_queue.cc:260] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:33.916208 24358 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:33.916934 23011 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 reported cstate change: config changed from index 9 to 10, 8d0f4e53cd9e469c9d5dffee5c42ea6c (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } attrs { replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.918190 24357 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7abd52761123466ca399914a88fb23aa (127.8.68.194) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.919811 24358 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b53d88de2b4e938f65b317b60b0630 in term 3.
I20240501 20:11:33.921343 23430 consensus_queue.cc:237] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } }
I20240501 20:11:33.929512 23155 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:11:33.929579 24357 raft_consensus.cc:1261] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Refusing update from remote peer ebb07a45c9604747986a01ae6f7f7bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:11:33.930157 25680 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:11:33.930701 25680 consensus_queue.cc:1035] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:11:33.933008 25680 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.933532 23156 leader_election.cc:288] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [CANDIDATE]: Term 3 election: Requested vote from peers ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603), 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689), 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196:33701)
I20240501 20:11:33.935678 23090 leader_election.cc:302] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 527d2c1d7b3d45aeb992646b89911c14, a4b53d88de2b4e938f65b317b60b0630, ebb07a45c9604747986a01ae6f7f7bc9; no voters: 
I20240501 20:11:33.936010 25054 raft_consensus.cc:2781] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:33.936611 25054 raft_consensus.cc:686] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 3 LEADER]: Becoming Leader. State: Replica: a4b53d88de2b4e938f65b317b60b0630, State: Running, Role: LEADER
I20240501 20:11:33.936941 25054 consensus_queue.cc:237] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:33.939528 23013 catalog_manager.cc:5484] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: term changed from 2 to 3, leader changed from 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) to a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193). New cstate: current_term: 3 leader_uuid: "a4b53d88de2b4e938f65b317b60b0630" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:33.940716 22999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c56eb0a04104909ae46e1c7c8d732c0 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:33.941321 23156 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.942502 23011 catalog_manager.cc:5484] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 reported cstate change: config changed from index 11 to 12, VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:33.945152 24357 raft_consensus.cc:2932] T 6c56eb0a04104909ae46e1c7c8d732c0 P 527d2c1d7b3d45aeb992646b89911c14 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.946254 25692 tablet_replica.cc:330] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa: stopping tablet replica
I20240501 20:11:33.946542 25692 raft_consensus.cc:2229] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:33.947204 25692 raft_consensus.cc:2258] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:33.948163 25692 ts_tablet_manager.cc:1875] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:33.948700 23293 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } }
I20240501 20:11:33.951226 24488 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:11:33.952345 23155 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:11:33.951763 25691 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240501 20:11:33.952713 24224 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:11:33.953085 25691 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240501 20:11:33.956364 25690 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240501 20:11:33.956903 25581 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.957535 23155 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.958086 24224 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.960669 23002 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7475f202d7f2476fbd69ef838d7e1ee2 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:33.961606 24488 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.964205 23011 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index 10 to 11, VOTER ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false replace: true } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.965746 24225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3d2792048e4b4aa59c034e0539ddb7" candidate_uuid: "a4b53d88de2b4e938f65b317b60b0630" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "5bd8f7fde1784f5fa1ab24113dc27934"
I20240501 20:11:33.966055 24225 raft_consensus.cc:3037] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:33.968925 24225 raft_consensus.cc:2454] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b53d88de2b4e938f65b317b60b0630 in term 3.
I20240501 20:11:33.973619 23293 consensus_queue.cc:237] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } }
I20240501 20:11:33.975536 24488 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:11:33.975834 23156 raft_consensus.cc:1261] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Refusing update from remote peer 7abd52761123466ca399914a88fb23aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:11:33.976079 25690 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:11:33.976460 25691 consensus_queue.cc:1035] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:11:33.976887 23409 tablet_service.cc:1514] Processing DeleteTablet for tablet 7475f202d7f2476fbd69ef838d7e1ee2 with delete_type TABLET_DATA_TOMBSTONED (TS ebb07a45c9604747986a01ae6f7f7bc9 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:60092
I20240501 20:11:33.978564 25581 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 7abd52761123466ca399914a88fb23aa [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.979132 24488 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P 8d0f4e53cd9e469c9d5dffee5c42ea6c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.979405 23155 raft_consensus.cc:2932] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.982653 25692 ts_tablet_manager.cc:1888] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 20:11:33.982859 23002 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7475f202d7f2476fbd69ef838d7e1ee2 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:33.983668 25692 log.cc:1196] T 6c56eb0a04104909ae46e1c7c8d732c0 P 7abd52761123466ca399914a88fb23aa: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/6c56eb0a04104909ae46e1c7c8d732c0
I20240501 20:11:33.984071 24204 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c56eb0a04104909ae46e1c7c8d732c0 with delete_type TABLET_DATA_TOMBSTONED (TS 5bd8f7fde1784f5fa1ab24113dc27934 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:37960
I20240501 20:11:33.984505 23002 catalog_manager.cc:4830] TS 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175): tablet 6c56eb0a04104909ae46e1c7c8d732c0 (table rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]) successfully deleted
I20240501 20:11:33.985164 23013 catalog_manager.cc:5484] T 7475f202d7f2476fbd69ef838d7e1ee2 P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index 11 to 12, VOTER 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196) evicted. New cstate: current_term: 1 leader_uuid: "7abd52761123466ca399914a88fb23aa" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } } peers { permanent_uuid: "8d0f4e53cd9e469c9d5dffee5c42ea6c" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 33911 } attrs { promote: false } } peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { promote: false } } }
I20240501 20:11:33.987238 25701 tablet_replica.cc:330] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: stopping tablet replica
I20240501 20:11:33.988155 25701 raft_consensus.cc:2229] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:33.989289 25699 tablet_replica.cc:330] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9: stopping tablet replica
I20240501 20:11:33.989495 25699 raft_consensus.cc:2229] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:33.989951 25699 raft_consensus.cc:2258] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:33.990206 24204 tablet_service.cc:1514] Processing DeleteTablet for tablet 7475f202d7f2476fbd69ef838d7e1ee2 with delete_type TABLET_DATA_TOMBSTONED (TS 5bd8f7fde1784f5fa1ab24113dc27934 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:37960
I20240501 20:11:33.991071 25701 raft_consensus.cc:2258] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:33.993108 25699 ts_tablet_manager.cc:1875] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:33.993113 25701 ts_tablet_manager.cc:1875] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:33.997782 25699 ts_tablet_manager.cc:1888] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 20:11:33.997885 25699 log.cc:1196] T 7475f202d7f2476fbd69ef838d7e1ee2 P ebb07a45c9604747986a01ae6f7f7bc9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/7475f202d7f2476fbd69ef838d7e1ee2
I20240501 20:11:33.998522 22999 catalog_manager.cc:4830] TS ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603): tablet 7475f202d7f2476fbd69ef838d7e1ee2 (table rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]) successfully deleted
I20240501 20:11:33.999323 25701 ts_tablet_manager.cc:1888] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 20:11:33.999461 25701 log.cc:1196] T 6c56eb0a04104909ae46e1c7c8d732c0 P 5bd8f7fde1784f5fa1ab24113dc27934: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/6c56eb0a04104909ae46e1c7c8d732c0
I20240501 20:11:34.000072 25701 tablet_replica.cc:330] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: stopping tablet replica
I20240501 20:11:34.000092 22999 catalog_manager.cc:4830] TS 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196:33701): tablet 6c56eb0a04104909ae46e1c7c8d732c0 (table rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]) successfully deleted
I20240501 20:11:34.000242 25701 raft_consensus.cc:2229] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:34.000528 25701 raft_consensus.cc:2258] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:34.001369 25701 ts_tablet_manager.cc:1875] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:34.006691 25701 ts_tablet_manager.cc:1888] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 20:11:34.006801 25701 log.cc:1196] T 7475f202d7f2476fbd69ef838d7e1ee2 P 5bd8f7fde1784f5fa1ab24113dc27934: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/wals/7475f202d7f2476fbd69ef838d7e1ee2
I20240501 20:11:34.007418 22999 catalog_manager.cc:4830] TS 5bd8f7fde1784f5fa1ab24113dc27934 (127.8.68.196:33701): tablet 7475f202d7f2476fbd69ef838d7e1ee2 (table rebalance_test_table_1 [id=0836886f6ba64deb8d3a9f1b1c2fb341]) successfully deleted
I20240501 20:11:34.045802 25549 raft_consensus.cc:1050] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14: attempting to promote NON_VOTER 7abd52761123466ca399914a88fb23aa to VOTER
I20240501 20:11:34.046718 25549 consensus_queue.cc:237] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } }
I20240501 20:11:34.049394 23156 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:34.049494 23293 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 LEARNER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:34.049949 25550 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:34.050228 25611 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:34.050448 23430 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:34.051321 25550 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:34.065694 25611 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7abd52761123466ca399914a88fb23aa (127.8.68.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } } }
I20240501 20:11:34.069061 23156 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7abd52761123466ca399914a88fb23aa (127.8.68.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } } }
I20240501 20:11:34.073583 23006 catalog_manager.cc:5484] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index 8 to 9, 7abd52761123466ca399914a88fb23aa (127.8.68.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "527d2c1d7b3d45aeb992646b89911c14" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } } }
I20240501 20:11:34.074404 23293 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7abd52761123466ca399914a88fb23aa (127.8.68.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } } }
I20240501 20:11:34.075302 23430 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7abd52761123466ca399914a88fb23aa (127.8.68.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } attrs { replace: true } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } } }
I20240501 20:11:34.085479 24358 consensus_queue.cc:237] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } }
I20240501 20:11:34.087090 23293 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:34.087395 23430 raft_consensus.cc:1261] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Refusing update from remote peer 527d2c1d7b3d45aeb992646b89911c14: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:34.087679 25746 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:34.088070 25611 consensus_queue.cc:1035] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:34.090034 25611 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } } }
I20240501 20:11:34.090660 23293 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } } }
I20240501 20:11:34.090904 23430 raft_consensus.cc:2932] T d940d4d4f1a54388b49f6794ca57d0df P ebb07a45c9604747986a01ae6f7f7bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } } }
I20240501 20:11:34.094310 23013 catalog_manager.cc:5484] T d940d4d4f1a54388b49f6794ca57d0df P 7abd52761123466ca399914a88fb23aa reported cstate change: config changed from index 9 to 10, VOTER a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "527d2c1d7b3d45aeb992646b89911c14" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false } } peers { permanent_uuid: "7abd52761123466ca399914a88fb23aa" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 41175 } attrs { promote: false } } }
I20240501 20:11:34.095701 22999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d940d4d4f1a54388b49f6794ca57d0df with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:34.097563 23135 tablet_service.cc:1514] Processing DeleteTablet for tablet d940d4d4f1a54388b49f6794ca57d0df with delete_type TABLET_DATA_TOMBSTONED (TS a4b53d88de2b4e938f65b317b60b0630 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41104
I20240501 20:11:34.098398 25747 tablet_replica.cc:330] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630: stopping tablet replica
I20240501 20:11:34.098573 25747 raft_consensus.cc:2229] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:34.098917 25747 raft_consensus.cc:2258] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:34.099647 25747 ts_tablet_manager.cc:1875] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:34.103763 25747 ts_tablet_manager.cc:1888] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:11:34.103871 25747 log.cc:1196] T d940d4d4f1a54388b49f6794ca57d0df P a4b53d88de2b4e938f65b317b60b0630: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/d940d4d4f1a54388b49f6794ca57d0df
I20240501 20:11:34.104481 23001 catalog_manager.cc:4830] TS a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789): tablet d940d4d4f1a54388b49f6794ca57d0df (table rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]) successfully deleted
I20240501 20:11:34.382647 24338 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:34.386512 23272 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:34.389276 24358 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 3 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 20:11:34.389868 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:34.389993 25054 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d2c1d7b3d45aeb992646b89911c14" member_type: VOTER last_known_addr { host: "127.8.68.198" port: 43689 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:34.400244 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:34.401811 23135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:34.417341 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:34.441817 23430 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 3 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 20:11:34.446363 25054 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20240501 20:11:34.474315 24224 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 3 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 20:11:34.477864 25054 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:34.901896 23156 consensus_queue.cc:237] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } }
I20240501 20:11:34.903672 24224 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 3 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 20:11:34.904042 23430 raft_consensus.cc:1261] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 3 FOLLOWER]: Refusing update from remote peer a4b53d88de2b4e938f65b317b60b0630: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 20:11:34.904444 25795 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:11:34.904819 25791 consensus_queue.cc:1035] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:11:34.906884 25795 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:34.907756 23430 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P ebb07a45c9604747986a01ae6f7f7bc9 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:34.910651 23001 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d3d2792048e4b4aa59c034e0539ddb7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:34.910611 24224 raft_consensus.cc:2932] T 3d3d2792048e4b4aa59c034e0539ddb7 P 5bd8f7fde1784f5fa1ab24113dc27934 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } } }
I20240501 20:11:34.912633 23013 catalog_manager.cc:5484] T 3d3d2792048e4b4aa59c034e0539ddb7 P a4b53d88de2b4e938f65b317b60b0630 reported cstate change: config changed from index 9 to 11, VOTER 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198) evicted. New cstate: current_term: 3 leader_uuid: "a4b53d88de2b4e938f65b317b60b0630" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a4b53d88de2b4e938f65b317b60b0630" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 39789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb07a45c9604747986a01ae6f7f7bc9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd8f7fde1784f5fa1ab24113dc27934" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 33701 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:34.917039 24338 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3d2792048e4b4aa59c034e0539ddb7 with delete_type TABLET_DATA_TOMBSTONED (TS 527d2c1d7b3d45aeb992646b89911c14 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44470
I20240501 20:11:34.917829 25811 tablet_replica.cc:330] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: stopping tablet replica
I20240501 20:11:34.918275 25811 raft_consensus.cc:2229] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:34.918579 25811 raft_consensus.cc:2258] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:34.919775 25811 ts_tablet_manager.cc:1875] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:34.924191 25811 ts_tablet_manager.cc:1888] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240501 20:11:34.924319 25811 log.cc:1196] T 3d3d2792048e4b4aa59c034e0539ddb7 P 527d2c1d7b3d45aeb992646b89911c14: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-5/wal/wals/3d3d2792048e4b4aa59c034e0539ddb7
I20240501 20:11:34.925017 22999 catalog_manager.cc:4830] TS 527d2c1d7b3d45aeb992646b89911c14 (127.8.68.198:43689): tablet 3d3d2792048e4b4aa59c034e0539ddb7 (table rebalance_test_table_2 [id=1b64bcc5481f44a49152f56c1f38c049]) successfully deleted
I20240501 20:11:35.012768 24338 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:35.016629 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:35.019454 23272 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:35.041568 23135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:35.055480 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:35.076232 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:35.199239 24204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:35.206599 24338 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:35.213989 23272 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:35.222007 23135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:35.224514 24468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:35.247387 23409 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:35.457405  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 23075
I20240501 20:11:35.468160 23197 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:35.607614  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 23075
W20240501 20:11:35.629225 23359 connection.cc:535] server connection from 127.8.68.193:53649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:35.629626  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 23208
I20240501 20:11:35.635164 23333 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:11:35.705534 23360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.193:39789: connect: Connection refused (error 111)
W20240501 20:11:35.707015 23360 consensus_peers.cc:485] T 6c56eb0a04104909ae46e1c7c8d732c0 P ebb07a45c9604747986a01ae6f7f7bc9 -> Peer a4b53d88de2b4e938f65b317b60b0630 (127.8.68.193:39789): Couldn't send request to peer a4b53d88de2b4e938f65b317b60b0630. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.193:39789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:35.779734  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 23208
I20240501 20:11:35.802346  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 23341
I20240501 20:11:35.808123 23471 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:11:35.862855 24293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.68.194:41175: connect: Connection refused (error 111)
W20240501 20:11:35.864140 24293 consensus_peers.cc:485] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 -> Peer 7abd52761123466ca399914a88fb23aa (127.8.68.194:41175): Couldn't send request to peer 7abd52761123466ca399914a88fb23aa. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.194:41175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:35.946020  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 23341
W20240501 20:11:35.968366 23001 connection.cc:535] server connection from 127.8.68.195:53023 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:35.968667  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 24138
I20240501 20:11:35.974473 24266 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:11:36.060842 24290 consensus_peers.cc:485] T d940d4d4f1a54388b49f6794ca57d0df P 527d2c1d7b3d45aeb992646b89911c14 -> Peer ebb07a45c9604747986a01ae6f7f7bc9 (127.8.68.195:45603): Couldn't send request to peer ebb07a45c9604747986a01ae6f7f7bc9. Status: Network error: Client connection negotiation failed: client connection to 127.8.68.195:45603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:36.088899  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 24138
I20240501 20:11:36.107372  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 24406
I20240501 20:11:36.113556 24529 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:36.220008  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 24406
I20240501 20:11:36.237365  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 24275
I20240501 20:11:36.248409 24398 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:36.375726  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 24275
I20240501 20:11:36.393977  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 22984
I20240501 20:11:36.395033 23045 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:36.538260  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 22984
2024-05-01T20:11:36Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3 (21718 ms)
[ RUN      ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20240501 20:11:36.590539  8467 test_util.cc:268] Using random seed: 402354343
I20240501 20:11:36.592190  8467 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:11:36.592264  8467 ts_itest-base.cc:113] --------------
I20240501 20:11:36.592301  8467 ts_itest-base.cc:114] 5 tablet servers
I20240501 20:11:36.592346  8467 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:11:36.592381  8467 ts_itest-base.cc:116] --------------
2024-05-01T20:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:11:36Z Disabled control of system clock
I20240501 20:11:36.634526  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:36731
--webserver_interface=127.8.68.254
--webserver_port=0
--builtin_ntp_servers=127.8.68.212:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.68.254:36731
--location_mapping_cmd=/tmp/dist-test-taskH_e3f3/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 20:11:36.939510 25902 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:11:36.939790 25902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:36.939888 25902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:36.939985 25902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:36.947077 25902 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:36.947180 25902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:11:36.947276 25902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:36.947331 25902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:11:36.947413 25902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:11:36.955787 25902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskH_e3f3/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.8.68.254:36731
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.68.254:36731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.68.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:36.956842 25902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:36.958382 25902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:36.968314 25907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:36.968750 25908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:36.969163 25910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:36.969679 25902 server_base.cc:1034] running on GCE node
I20240501 20:11:36.970315 25902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:36.971473 25902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:36.972716 25902 hybrid_clock.cc:648] HybridClock initialized: now 1714594296972648 us; error 39 us; skew 500 ppm
I20240501 20:11:36.973038 25902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:36.975219 25902 webserver.cc:460] Webserver started at http://127.8.68.254:33013/ using document root <none> and password file <none>
I20240501 20:11:36.975888 25902 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:36.976023 25902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:36.976323 25902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:36.980413 25902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "88a8ae526e9845058885f6ff8718333b"
format_stamp: "Formatted at 2024-05-01 20:11:36 on dist-test-slave-08p9"
I20240501 20:11:36.981197 25902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "88a8ae526e9845058885f6ff8718333b"
format_stamp: "Formatted at 2024-05-01 20:11:36 on dist-test-slave-08p9"
I20240501 20:11:36.986104 25902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.005s
I20240501 20:11:36.989096 25916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:36.990236 25902 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:36.990411 25902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
uuid: "88a8ae526e9845058885f6ff8718333b"
format_stamp: "Formatted at 2024-05-01 20:11:36 on dist-test-slave-08p9"
I20240501 20:11:36.990561 25902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:37.007817 25902 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:37.008533 25902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:37.008837 25902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:37.025216 25902 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.254:36731
I20240501 20:11:37.025245 25968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.254:36731 every 8 connection(s)
I20240501 20:11:37.026415 25902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:11:37.029600 25969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:37.034976  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 25902
I20240501 20:11:37.035177  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:11:37.038508 25969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b: Bootstrap starting.
I20240501 20:11:37.042135 25969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:37.043349 25969 log.cc:824] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:37.045801 25969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b: No bootstrap required, opened a new log
I20240501 20:11:37.052102 25969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a8ae526e9845058885f6ff8718333b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 36731 } }
I20240501 20:11:37.052366 25969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:37.052436 25969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a8ae526e9845058885f6ff8718333b, State: Initialized, Role: FOLLOWER
I20240501 20:11:37.052888 25969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a8ae526e9845058885f6ff8718333b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 36731 } }
I20240501 20:11:37.053056 25969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:37.053144 25969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:37.053256 25969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:37.055088 25969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a8ae526e9845058885f6ff8718333b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 36731 } }
I20240501 20:11:37.055502 25969 leader_election.cc:302] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88a8ae526e9845058885f6ff8718333b; no voters: 
I20240501 20:11:37.056087 25969 leader_election.cc:288] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 20:11:37.056239 25974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:37.056838 25974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [term 1 LEADER]: Becoming Leader. State: Replica: 88a8ae526e9845058885f6ff8718333b, State: Running, Role: LEADER
I20240501 20:11:37.057377 25974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a8ae526e9845058885f6ff8718333b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 36731 } }
I20240501 20:11:37.057744 25969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:11:37.059733 25976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88a8ae526e9845058885f6ff8718333b. Latest consensus state: current_term: 1 leader_uuid: "88a8ae526e9845058885f6ff8718333b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a8ae526e9845058885f6ff8718333b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 36731 } } }
I20240501 20:11:37.060011 25976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:37.061650 25979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:11:37.061756 25975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88a8ae526e9845058885f6ff8718333b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a8ae526e9845058885f6ff8718333b" member_type: VOTER last_known_addr { host: "127.8.68.254" port: 36731 } } }
I20240501 20:11:37.062057 25975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:37.066777 25979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:11:37.074822 25979 catalog_manager.cc:1347] Generated new cluster ID: 40295282822645309b10f145da6adaa7
I20240501 20:11:37.074932 25979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:11:37.121225 25979 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:11:37.122249 25979 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:11:37.140029 25979 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 88a8ae526e9845058885f6ff8718333b: Generated new TSK 0
I20240501 20:11:37.140697 25979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:11:37.154824  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.193:0
--local_ip_for_outbound_sockets=127.8.68.193
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--builtin_ntp_servers=127.8.68.212:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:37.470611 25994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:37.470849 25994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:37.470947 25994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:37.477942 25994 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:37.478039 25994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:37.478154 25994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.193
I20240501 20:11:37.486539 25994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.68.193
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.193
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:37.487568 25994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:37.489089 25994 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:37.500242 25999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:37.500679 26000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:37.501164 26002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:37.501659 25994 server_base.cc:1034] running on GCE node
I20240501 20:11:37.502142 25994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:37.502969 25994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:37.504174 25994 hybrid_clock.cc:648] HybridClock initialized: now 1714594297504138 us; error 25 us; skew 500 ppm
I20240501 20:11:37.504456 25994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:37.506479 25994 webserver.cc:460] Webserver started at http://127.8.68.193:37905/ using document root <none> and password file <none>
I20240501 20:11:37.507118 25994 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:37.507216 25994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:37.507493 25994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:37.509752 25994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
format_stamp: "Formatted at 2024-05-01 20:11:37 on dist-test-slave-08p9"
I20240501 20:11:37.510474 25994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
format_stamp: "Formatted at 2024-05-01 20:11:37 on dist-test-slave-08p9"
I20240501 20:11:37.514907 25994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240501 20:11:37.517855 26009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:37.519002 25994 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:11:37.519142 25994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
format_stamp: "Formatted at 2024-05-01 20:11:37 on dist-test-slave-08p9"
I20240501 20:11:37.519305 25994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:37.538749 25994 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:37.539462 25994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:37.539723 25994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:37.540594 25994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:37.542032 25994 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:37.542131 25994 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:37.542244 25994 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:37.542346 25994 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.001s
I20240501 20:11:37.571254 25994 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.193:41067
I20240501 20:11:37.571321 26123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.193:41067 every 8 connection(s)
I20240501 20:11:37.572363 25994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:11:37.581907  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 25994
I20240501 20:11:37.582113  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:11:37.583731 26124 heartbeater.cc:347] Connected to a master server at 127.8.68.254:36731
I20240501 20:11:37.584005 26124 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:37.584831 26124 heartbeater.cc:511] Master 127.8.68.254:36731 requested a full tablet report, sending...
I20240501 20:11:37.585423  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.194:0
--local_ip_for_outbound_sockets=127.8.68.194
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--builtin_ntp_servers=127.8.68.212:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:37.630046 25933 ts_manager.cc:194] Registered new tserver with Master: e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067)
I20240501 20:11:37.632462 25933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.193:36285
W20240501 20:11:37.893651 26130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:37.893882 26130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:37.893977 26130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:37.901199 26130 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:37.901293 26130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:37.901403 26130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.194
I20240501 20:11:37.909667 26130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.68.194
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.194
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:37.910719 26130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:37.912189 26130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:37.923506 26136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:37.923902 26135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:37.924336 26138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:37.925019 26130 server_base.cc:1034] running on GCE node
I20240501 20:11:37.925513 26130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:37.926340 26130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:37.927590 26130 hybrid_clock.cc:648] HybridClock initialized: now 1714594297927550 us; error 40 us; skew 500 ppm
I20240501 20:11:37.927866 26130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:37.929805 26130 webserver.cc:460] Webserver started at http://127.8.68.194:33675/ using document root <none> and password file <none>
I20240501 20:11:37.930478 26130 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:37.930584 26130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:37.930860 26130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:37.933094 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "41abd2deb0d7412bbb66b1aa9851d888"
format_stamp: "Formatted at 2024-05-01 20:11:37 on dist-test-slave-08p9"
I20240501 20:11:37.933789 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "41abd2deb0d7412bbb66b1aa9851d888"
format_stamp: "Formatted at 2024-05-01 20:11:37 on dist-test-slave-08p9"
I20240501 20:11:37.938143 26130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20240501 20:11:37.941061 26144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:37.942165 26130 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:37.942338 26130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "41abd2deb0d7412bbb66b1aa9851d888"
format_stamp: "Formatted at 2024-05-01 20:11:37 on dist-test-slave-08p9"
I20240501 20:11:37.942507 26130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:37.960937 26130 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:37.961587 26130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:37.961812 26130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:37.962664 26130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:37.963997 26130 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:37.964087 26130 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:37.964190 26130 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:37.964255 26130 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:37.992033 26130 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.194:35909
I20240501 20:11:37.992090 26257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.194:35909 every 8 connection(s)
I20240501 20:11:37.993104 26130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:11:38.003031  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 26130
I20240501 20:11:38.003217  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:11:38.005496 26258 heartbeater.cc:347] Connected to a master server at 127.8.68.254:36731
I20240501 20:11:38.005779 26258 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:38.006381  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.195:0
--local_ip_for_outbound_sockets=127.8.68.195
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--builtin_ntp_servers=127.8.68.212:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:38.006553 26258 heartbeater.cc:511] Master 127.8.68.254:36731 requested a full tablet report, sending...
I20240501 20:11:38.057097 25933 ts_manager.cc:194] Registered new tserver with Master: 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:38.058355 25933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.194:39267
W20240501 20:11:38.321714 26262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:38.321954 26262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:38.322043 26262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:38.329154 26262 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:38.329249 26262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:38.329365 26262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.195
I20240501 20:11:38.337659 26262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.68.195
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.195
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:38.338773 26262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:38.340299 26262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:38.351210 26268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:38.351735 26271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:38.352205 26269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:38.352720 26262 server_base.cc:1034] running on GCE node
I20240501 20:11:38.353222 26262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:38.354033 26262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:38.355216 26262 hybrid_clock.cc:648] HybridClock initialized: now 1714594298355176 us; error 30 us; skew 500 ppm
I20240501 20:11:38.355487 26262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:38.357457 26262 webserver.cc:460] Webserver started at http://127.8.68.195:41571/ using document root <none> and password file <none>
I20240501 20:11:38.358127 26262 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:38.358239 26262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:38.358558 26262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:38.360913 26262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
format_stamp: "Formatted at 2024-05-01 20:11:38 on dist-test-slave-08p9"
I20240501 20:11:38.361617 26262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
format_stamp: "Formatted at 2024-05-01 20:11:38 on dist-test-slave-08p9"
I20240501 20:11:38.366600 26262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240501 20:11:38.369534 26277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:38.370728 26262 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:38.370859 26262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
format_stamp: "Formatted at 2024-05-01 20:11:38 on dist-test-slave-08p9"
I20240501 20:11:38.371027 26262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:38.390336 26262 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:38.391012 26262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:38.391233 26262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:38.392073 26262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:38.393368 26262 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:38.393450 26262 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:38.393545 26262 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:38.393636 26262 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:38.425113 26262 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.195:37745
I20240501 20:11:38.425191 26389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.195:37745 every 8 connection(s)
I20240501 20:11:38.426192 26262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:11:38.433493  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 26262
I20240501 20:11:38.433650  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:11:38.437042  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:0
--local_ip_for_outbound_sockets=127.8.68.196
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--builtin_ntp_servers=127.8.68.212:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:38.437561 26390 heartbeater.cc:347] Connected to a master server at 127.8.68.254:36731
I20240501 20:11:38.437793 26390 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:38.438516 26390 heartbeater.cc:511] Master 127.8.68.254:36731 requested a full tablet report, sending...
I20240501 20:11:38.494542 25933 ts_manager.cc:194] Registered new tserver with Master: 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:38.495589 25933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.195:44733
I20240501 20:11:38.635504 26124 heartbeater.cc:503] Master 127.8.68.254:36731 was elected leader, sending a full tablet report...
W20240501 20:11:38.765100 26394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:38.765334 26394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:38.765455 26394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:38.772495 26394 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:38.772589 26394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:38.772733 26394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:11:38.780974 26394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:38.781991 26394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:38.783550 26394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:38.794443 26399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:38.794952 26402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:38.795405 26400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:38.795918 26394 server_base.cc:1034] running on GCE node
I20240501 20:11:38.796415 26394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:38.797188 26394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:38.798429 26394 hybrid_clock.cc:648] HybridClock initialized: now 1714594298798368 us; error 43 us; skew 500 ppm
I20240501 20:11:38.798765 26394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:38.800871 26394 webserver.cc:460] Webserver started at http://127.8.68.196:45127/ using document root <none> and password file <none>
I20240501 20:11:38.801549 26394 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:38.801651 26394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:38.801920 26394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:38.804405 26394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0afaccbf1fb84dd7b530d6632c4cd948"
format_stamp: "Formatted at 2024-05-01 20:11:38 on dist-test-slave-08p9"
I20240501 20:11:38.805078 26394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0afaccbf1fb84dd7b530d6632c4cd948"
format_stamp: "Formatted at 2024-05-01 20:11:38 on dist-test-slave-08p9"
I20240501 20:11:38.809527 26394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20240501 20:11:38.812561 26408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:38.813686 26394 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:38.813819 26394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0afaccbf1fb84dd7b530d6632c4cd948"
format_stamp: "Formatted at 2024-05-01 20:11:38 on dist-test-slave-08p9"
I20240501 20:11:38.813993 26394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:38.827154 26394 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:38.827836 26394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:38.828063 26394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:38.828869 26394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:38.830137 26394 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:38.830230 26394 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:38.830348 26394 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:38.830415 26394 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:38.857340 26394 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:34631
I20240501 20:11:38.857393 26522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:34631 every 8 connection(s)
I20240501 20:11:38.858521 26394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:38.862069  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 26394
I20240501 20:11:38.862250  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:11:38.865599  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:0
--local_ip_for_outbound_sockets=127.8.68.197
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--builtin_ntp_servers=127.8.68.212:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:38.870414 26523 heartbeater.cc:347] Connected to a master server at 127.8.68.254:36731
I20240501 20:11:38.870673 26523 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:38.871369 26523 heartbeater.cc:511] Master 127.8.68.254:36731 requested a full tablet report, sending...
I20240501 20:11:38.927719 25933 ts_manager.cc:194] Registered new tserver with Master: 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631)
I20240501 20:11:38.928656 25933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:36433
I20240501 20:11:39.060868 26258 heartbeater.cc:503] Master 127.8.68.254:36731 was elected leader, sending a full tablet report...
W20240501 20:11:39.192802 26526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:39.193037 26526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:39.193142 26526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:39.202487 26526 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:39.202589 26526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:39.202700 26526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:11:39.211083 26526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=0
--tserver_master_addrs=127.8.68.254:36731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:39.212127 26526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:39.213639 26526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:39.224920 26532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:39.225378 26533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:39.225761 26535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:39.226377 26526 server_base.cc:1034] running on GCE node
I20240501 20:11:39.226886 26526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:39.227730 26526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:39.228929 26526 hybrid_clock.cc:648] HybridClock initialized: now 1714594299228887 us; error 38 us; skew 500 ppm
I20240501 20:11:39.229213 26526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:39.231287 26526 webserver.cc:460] Webserver started at http://127.8.68.197:38925/ using document root <none> and password file <none>
I20240501 20:11:39.231913 26526 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:39.232018 26526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:39.232334 26526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:39.234787 26526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f79d4befcb2c4d6cb0e1b79379164c25"
format_stamp: "Formatted at 2024-05-01 20:11:39 on dist-test-slave-08p9"
I20240501 20:11:39.235504 26526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f79d4befcb2c4d6cb0e1b79379164c25"
format_stamp: "Formatted at 2024-05-01 20:11:39 on dist-test-slave-08p9"
I20240501 20:11:39.240036 26526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.002s
I20240501 20:11:39.243022 26541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:39.244165 26526 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 20:11:39.244303 26526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f79d4befcb2c4d6cb0e1b79379164c25"
format_stamp: "Formatted at 2024-05-01 20:11:39 on dist-test-slave-08p9"
I20240501 20:11:39.244474 26526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:39.260841 26526 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:39.261485 26526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:39.261718 26526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:39.262578 26526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:39.263936 26526 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:39.264016 26526 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:39.264091 26526 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:39.264142 26526 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:39.291963 26526 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:46545
I20240501 20:11:39.292021 26653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:46545 every 8 connection(s)
I20240501 20:11:39.293076 26526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:11:39.298815  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 26526
I20240501 20:11:39.299005  8467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:11:39.304890 26654 heartbeater.cc:347] Connected to a master server at 127.8.68.254:36731
I20240501 20:11:39.305120 26654 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:39.305794 26654 heartbeater.cc:511] Master 127.8.68.254:36731 requested a full tablet report, sending...
I20240501 20:11:39.347440 25933 ts_manager.cc:194] Registered new tserver with Master: f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545)
I20240501 20:11:39.348394 25933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:60803
I20240501 20:11:39.350939  8467 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:11:39.366423  8467 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:11:39.366559  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 26526
I20240501 20:11:39.373803 26649 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:39.454387  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 26526
I20240501 20:11:39.467623  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 26394
I20240501 20:11:39.475332 26518 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:39.498091 26390 heartbeater.cc:503] Master 127.8.68.254:36731 was elected leader, sending a full tablet report...
I20240501 20:11:39.554181  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 26394
I20240501 20:11:43.319988 25933 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41972:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 20:11:43.326700 25933 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:11:43.353706 26325 tablet_service.cc:1467] Processing CreateTablet for tablet f35f4b0292e045949cae7b7f4120b67c (DEFAULT_TABLE table=rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:43.355185 26325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35f4b0292e045949cae7b7f4120b67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:43.354851 26057 tablet_service.cc:1467] Processing CreateTablet for tablet a5c251d108ea4e7f85f56173c24fdec4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:43.355038 26193 tablet_service.cc:1467] Processing CreateTablet for tablet f35f4b0292e045949cae7b7f4120b67c (DEFAULT_TABLE table=rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:43.355937 26057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c251d108ea4e7f85f56173c24fdec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:43.356302 26193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35f4b0292e045949cae7b7f4120b67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:43.358765 26191 tablet_service.cc:1467] Processing CreateTablet for tablet cd17acc8002948999a5c607ccb9bb189 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:43.358796 26058 tablet_service.cc:1467] Processing CreateTablet for tablet f35f4b0292e045949cae7b7f4120b67c (DEFAULT_TABLE table=rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:43.358959 26056 tablet_service.cc:1467] Processing CreateTablet for tablet cd17acc8002948999a5c607ccb9bb189 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:43.359180 26191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd17acc8002948999a5c607ccb9bb189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:43.359583 26192 tablet_service.cc:1467] Processing CreateTablet for tablet a5c251d108ea4e7f85f56173c24fdec4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:43.359915 26058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35f4b0292e045949cae7b7f4120b67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:43.359793 26324 tablet_service.cc:1467] Processing CreateTablet for tablet a5c251d108ea4e7f85f56173c24fdec4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:43.359978 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c251d108ea4e7f85f56173c24fdec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:43.359869 26323 tablet_service.cc:1467] Processing CreateTablet for tablet cd17acc8002948999a5c607ccb9bb189 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:43.360245 26324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c251d108ea4e7f85f56173c24fdec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:43.360630 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd17acc8002948999a5c607ccb9bb189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:43.360743 26056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd17acc8002948999a5c607ccb9bb189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:43.375087 26678 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: Bootstrap starting.
I20240501 20:11:43.376075 26679 tablet_bootstrap.cc:492] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888: Bootstrap starting.
I20240501 20:11:43.378558 26678 tablet_bootstrap.cc:654] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:43.379815 26678 log.cc:824] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:43.380306 26680 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: Bootstrap starting.
I20240501 20:11:43.380993 26679 tablet_bootstrap.cc:654] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:43.382304 26679 log.cc:824] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:43.383592 26680 tablet_bootstrap.cc:654] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:43.384832 26680 log.cc:824] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:43.388867 26678 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: No bootstrap required, opened a new log
I20240501 20:11:43.389175 26678 ts_tablet_manager.cc:1367] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.005s
I20240501 20:11:43.389226 26679 tablet_bootstrap.cc:492] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888: No bootstrap required, opened a new log
I20240501 20:11:43.389225 26680 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: No bootstrap required, opened a new log
I20240501 20:11:43.389447 26680 ts_tablet_manager.cc:1367] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240501 20:11:43.389458 26679 ts_tablet_manager.cc:1367] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.007s
I20240501 20:11:43.395141 26680 raft_consensus.cc:348] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.395506 26680 raft_consensus.cc:374] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:43.395597 26680 raft_consensus.cc:729] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Initialized, Role: FOLLOWER
I20240501 20:11:43.396165 26680 consensus_queue.cc:260] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.397678 26679 raft_consensus.cc:348] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.397675 26678 raft_consensus.cc:348] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } }
I20240501 20:11:43.397948 26680 ts_tablet_manager.cc:1398] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.004s
I20240501 20:11:43.398051 26678 raft_consensus.cc:374] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:43.398082 26679 raft_consensus.cc:374] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:43.398185 26679 raft_consensus.cc:729] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Initialized, Role: FOLLOWER
I20240501 20:11:43.398182 26678 raft_consensus.cc:729] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:43.398591 26680 tablet_bootstrap.cc:492] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9: Bootstrap starting.
I20240501 20:11:43.398849 26679 consensus_queue.cc:260] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.398954 26678 consensus_queue.cc:260] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } }
I20240501 20:11:43.400019 26680 tablet_bootstrap.cc:654] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:43.403820 26678 ts_tablet_manager.cc:1398] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: Time spent starting tablet: real 0.014s	user 0.010s	sys 0.003s
I20240501 20:11:43.405496 26678 tablet_bootstrap.cc:492] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: Bootstrap starting.
I20240501 20:11:43.407187 26678 tablet_bootstrap.cc:654] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:43.409406 26679 ts_tablet_manager.cc:1398] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.013s
I20240501 20:11:43.409839 26679 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888: Bootstrap starting.
I20240501 20:11:43.410929 26678 tablet_bootstrap.cc:492] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: No bootstrap required, opened a new log
I20240501 20:11:43.411111 26678 ts_tablet_manager.cc:1367] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 20:11:43.411443 26679 tablet_bootstrap.cc:654] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:43.412084 26678 raft_consensus.cc:348] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.412348 26678 raft_consensus.cc:374] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:43.412482 26678 raft_consensus.cc:729] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:43.412794 26678 consensus_queue.cc:260] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.412971 26680 tablet_bootstrap.cc:492] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9: No bootstrap required, opened a new log
I20240501 20:11:43.413139 26680 ts_tablet_manager.cc:1367] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.004s
I20240501 20:11:43.413324 26679 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888: No bootstrap required, opened a new log
I20240501 20:11:43.413379 26678 ts_tablet_manager.cc:1398] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:11:43.413481 26679 ts_tablet_manager.cc:1367] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:11:43.413631 26678 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0: Bootstrap starting.
I20240501 20:11:43.414032 26680 raft_consensus.cc:348] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.414254 26680 raft_consensus.cc:374] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:43.414397 26680 raft_consensus.cc:729] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Initialized, Role: FOLLOWER
I20240501 20:11:43.414430 26679 raft_consensus.cc:348] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } }
I20240501 20:11:43.414634 26679 raft_consensus.cc:374] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:43.414624 26680 consensus_queue.cc:260] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.414767 26679 raft_consensus.cc:729] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Initialized, Role: FOLLOWER
I20240501 20:11:43.415055 26679 consensus_queue.cc:260] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } }
I20240501 20:11:43.415233 26678 tablet_bootstrap.cc:654] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:43.415539 26679 ts_tablet_manager.cc:1398] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:43.415747 26680 ts_tablet_manager.cc:1398] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:43.415807 26679 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888: Bootstrap starting.
I20240501 20:11:43.416008 26680 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9: Bootstrap starting.
I20240501 20:11:43.417263 26679 tablet_bootstrap.cc:654] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:43.417538 26680 tablet_bootstrap.cc:654] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:43.417927 26687 raft_consensus.cc:482] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:43.418125 26687 raft_consensus.cc:504] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } }
I20240501 20:11:43.419997 26687 leader_election.cc:288] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745), 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:43.422478 26680 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9: No bootstrap required, opened a new log
I20240501 20:11:43.422709 26680 ts_tablet_manager.cc:1367] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240501 20:11:43.423310 26679 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888: No bootstrap required, opened a new log
I20240501 20:11:43.423444 26679 ts_tablet_manager.cc:1367] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.002s
I20240501 20:11:43.423345 26680 raft_consensus.cc:348] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } }
I20240501 20:11:43.423552 26680 raft_consensus.cc:374] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:43.423630 26680 raft_consensus.cc:729] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Initialized, Role: FOLLOWER
I20240501 20:11:43.423849 26680 consensus_queue.cc:260] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } }
I20240501 20:11:43.424005 26679 raft_consensus.cc:348] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.424249 26679 raft_consensus.cc:374] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:43.424345 26679 raft_consensus.cc:729] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Initialized, Role: FOLLOWER
I20240501 20:11:43.424376 26680 ts_tablet_manager.cc:1398] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:11:43.424590 26679 consensus_queue.cc:260] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.425663 26679 ts_tablet_manager.cc:1398] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
W20240501 20:11:43.430390 26391 tablet.cc:2369] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:11:43.436766 26678 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0: No bootstrap required, opened a new log
I20240501 20:11:43.436910 26678 ts_tablet_manager.cc:1367] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0: Time spent bootstrapping tablet: real 0.023s	user 0.008s	sys 0.000s
I20240501 20:11:43.437539 26678 raft_consensus.cc:348] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.437764 26678 raft_consensus.cc:374] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:43.437849 26678 raft_consensus.cc:729] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:43.438083 26678 consensus_queue.cc:260] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.439064 26678 ts_tablet_manager.cc:1398] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:43.438912 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c" candidate_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41abd2deb0d7412bbb66b1aa9851d888" is_pre_election: true
I20240501 20:11:43.439386 26213 raft_consensus.cc:2454] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a0d830efeb4d43b13221b36d8e77f0 in term 0.
I20240501 20:11:43.439920 26013 leader_election.cc:302] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41abd2deb0d7412bbb66b1aa9851d888, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:43.440264 26687 raft_consensus.cc:2781] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:43.440389 26687 raft_consensus.cc:482] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:43.440490 26687 raft_consensus.cc:3037] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:43.441012 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c" candidate_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" is_pre_election: true
I20240501 20:11:43.441493 26345 raft_consensus.cc:2454] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a0d830efeb4d43b13221b36d8e77f0 in term 0.
I20240501 20:11:43.442505 26687 raft_consensus.cc:504] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } }
I20240501 20:11:43.443017 26687 leader_election.cc:288] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [CANDIDATE]: Term 1 election: Requested vote from peers 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745), 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:43.443358 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c" candidate_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
I20240501 20:11:43.443521 26345 raft_consensus.cc:3037] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:43.443529 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c" candidate_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41abd2deb0d7412bbb66b1aa9851d888"
I20240501 20:11:43.443672 26213 raft_consensus.cc:3037] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:43.445360 26345 raft_consensus.cc:2454] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a0d830efeb4d43b13221b36d8e77f0 in term 1.
I20240501 20:11:43.445502 26213 raft_consensus.cc:2454] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a0d830efeb4d43b13221b36d8e77f0 in term 1.
I20240501 20:11:43.445693 26012 leader_election.cc:302] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:43.445941 26687 raft_consensus.cc:2781] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:43.446342 26687 raft_consensus.cc:686] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Becoming Leader. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Running, Role: LEADER
I20240501 20:11:43.446816 26687 consensus_queue.cc:237] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } }
I20240501 20:11:43.450697 25933 catalog_manager.cc:5484] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193). New cstate: current_term: 1 leader_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:43.494170 26685 raft_consensus.cc:482] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:43.494338 26685 raft_consensus.cc:504] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.495790 26685 leader_election.cc:288] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909), e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067)
W20240501 20:11:43.497764 26259 tablet.cc:2369] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:11:43.502573 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c251d108ea4e7f85f56173c24fdec4" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41abd2deb0d7412bbb66b1aa9851d888" is_pre_election: true
I20240501 20:11:43.502782 26213 raft_consensus.cc:2454] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 0.
I20240501 20:11:43.503202 26281 leader_election.cc:302] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, 41abd2deb0d7412bbb66b1aa9851d888; no voters: 
I20240501 20:11:43.503286 26078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c251d108ea4e7f85f56173c24fdec4" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" is_pre_election: true
I20240501 20:11:43.503563 26685 raft_consensus.cc:2781] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:43.503628 26078 raft_consensus.cc:2454] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 0.
I20240501 20:11:43.503674 26685 raft_consensus.cc:482] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:43.503777 26685 raft_consensus.cc:3037] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:43.505568 26685 raft_consensus.cc:504] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.506078 26685 leader_election.cc:288] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 election: Requested vote from peers 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909), e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067)
I20240501 20:11:43.506513 26078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c251d108ea4e7f85f56173c24fdec4" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
I20240501 20:11:43.506652 26078 raft_consensus.cc:3037] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:43.506696 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c251d108ea4e7f85f56173c24fdec4" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41abd2deb0d7412bbb66b1aa9851d888"
I20240501 20:11:43.506893 26213 raft_consensus.cc:3037] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:43.508594 26078 raft_consensus.cc:2454] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 1.
I20240501 20:11:43.508631 26213 raft_consensus.cc:2454] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 1.
I20240501 20:11:43.508927 26281 leader_election.cc:302] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:43.509187 26685 raft_consensus.cc:2781] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:43.509562 26685 raft_consensus.cc:686] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Running, Role: LEADER
I20240501 20:11:43.510005 26685 consensus_queue.cc:237] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.513883 25933 catalog_manager.cc:5484] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:43.550444 26686 raft_consensus.cc:482] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:43.550576 26686 raft_consensus.cc:504] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.551848 26686 leader_election.cc:288] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:43.559412 26078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd17acc8002948999a5c607ccb9bb189" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" is_pre_election: true
I20240501 20:11:43.559464 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd17acc8002948999a5c607ccb9bb189" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" is_pre_election: true
I20240501 20:11:43.559638 26078 raft_consensus.cc:2454] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:43.559648 26345 raft_consensus.cc:2454] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:43.560082 26148 leader_election.cc:302] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41abd2deb0d7412bbb66b1aa9851d888, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:43.560424 26686 raft_consensus.cc:2781] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:43.560532 26686 raft_consensus.cc:482] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:43.560609 26686 raft_consensus.cc:3037] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:43.562448 26686 raft_consensus.cc:504] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.563007 26686 leader_election.cc:288] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Requested vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:43.563303 26078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd17acc8002948999a5c607ccb9bb189" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
I20240501 20:11:43.563375 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd17acc8002948999a5c607ccb9bb189" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
I20240501 20:11:43.563470 26078 raft_consensus.cc:3037] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:43.563550 26345 raft_consensus.cc:3037] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:43.565246 26078 raft_consensus.cc:2454] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:43.565271 26345 raft_consensus.cc:2454] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:43.565579 26148 leader_election.cc:302] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41abd2deb0d7412bbb66b1aa9851d888, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:43.565834 26686 raft_consensus.cc:2781] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:43.566193 26686 raft_consensus.cc:686] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Becoming Leader. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Running, Role: LEADER
I20240501 20:11:43.566617 26686 consensus_queue.cc:237] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:43.570214 25933 catalog_manager.cc:5484] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:43.577018 26125 tablet.cc:2369] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:11:43.892555 26706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:43.892822 26706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:43.899845 26706 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 20:11:44.013231 26731 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:44.046933 26345 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:44.047814 26686 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:44.049763 26078 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:44.050444 26686 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:44.065409 26739 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:44.075106 26741 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:44.075908 26741 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:44.094843 26737 mvcc.cc:205] Tried to move back new op lower bound from 7022978269385011200 to 7022978266921672704. Current Snapshot: MvccSnapshot[applied={T|T < 7022978269385011200}]
I20240501 20:11:44.147393 26745 mvcc.cc:205] Tried to move back new op lower bound from 7022978269600034816 to 7022978267180544000. Current Snapshot: MvccSnapshot[applied={T|T < 7022978269600034816}]
W20240501 20:11:44.198446 26148 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.68.193:41067, user_credentials={real_user=slave}} blocked reactor thread for 49245.1us

INSERT report
    rows total: 2000
    time total: 265.844 ms
  time per row: 0.132922 ms
I20240501 20:11:44.302129 25931 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41972:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 20:11:44.302971 25931 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:11:44.310665 26058 tablet_service.cc:1467] Processing CreateTablet for tablet e3d4fc9946d44e548a0bac91834b8afd (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:44.311100 26325 tablet_service.cc:1467] Processing CreateTablet for tablet e3d4fc9946d44e548a0bac91834b8afd (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:44.311254 26058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d4fc9946d44e548a0bac91834b8afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:44.311509 26325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d4fc9946d44e548a0bac91834b8afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:44.312103 26056 tablet_service.cc:1467] Processing CreateTablet for tablet 57621c56ad05419aaafd7036ced2b691 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:44.312211 26057 tablet_service.cc:1467] Processing CreateTablet for tablet 97f145d7f939459fbc804b732189db3f (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:44.312482 26056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57621c56ad05419aaafd7036ced2b691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:44.312928 26192 tablet_service.cc:1467] Processing CreateTablet for tablet 57621c56ad05419aaafd7036ced2b691 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:44.313324 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57621c56ad05419aaafd7036ced2b691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:44.314473 26057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f145d7f939459fbc804b732189db3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:44.318631 26193 tablet_service.cc:1467] Processing CreateTablet for tablet 97f145d7f939459fbc804b732189db3f (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:44.318993 26193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f145d7f939459fbc804b732189db3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:44.323333 26323 tablet_service.cc:1467] Processing CreateTablet for tablet 57621c56ad05419aaafd7036ced2b691 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:44.323721 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57621c56ad05419aaafd7036ced2b691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:44.324491 26775 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0: Bootstrap starting.
I20240501 20:11:44.325786 26774 tablet_bootstrap.cc:492] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9: Bootstrap starting.
I20240501 20:11:44.326212 26775 tablet_bootstrap.cc:654] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:44.326661 26324 tablet_service.cc:1467] Processing CreateTablet for tablet 97f145d7f939459fbc804b732189db3f (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:44.327116 26324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f145d7f939459fbc804b732189db3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:44.327549 26191 tablet_service.cc:1467] Processing CreateTablet for tablet e3d4fc9946d44e548a0bac91834b8afd (DEFAULT_TABLE table=rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:44.327771 26774 tablet_bootstrap.cc:654] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:44.327903 26191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d4fc9946d44e548a0bac91834b8afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:44.330003 26774 tablet_bootstrap.cc:492] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9: No bootstrap required, opened a new log
I20240501 20:11:44.330202 26774 ts_tablet_manager.cc:1367] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 20:11:44.331194 26774 raft_consensus.cc:348] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.331442 26774 raft_consensus.cc:374] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:44.331596 26774 raft_consensus.cc:729] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Initialized, Role: FOLLOWER
I20240501 20:11:44.331907 26774 consensus_queue.cc:260] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.332562 26774 ts_tablet_manager.cc:1398] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:44.336221 26774 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9: Bootstrap starting.
I20240501 20:11:44.338088 26774 tablet_bootstrap.cc:654] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:44.338408 26775 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0: No bootstrap required, opened a new log
I20240501 20:11:44.338591 26775 ts_tablet_manager.cc:1367] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.000s
I20240501 20:11:44.339421 26775 raft_consensus.cc:348] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.339624 26775 raft_consensus.cc:374] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:44.339740 26775 raft_consensus.cc:729] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:44.339948 26774 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9: No bootstrap required, opened a new log
I20240501 20:11:44.340098 26774 ts_tablet_manager.cc:1367] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240501 20:11:44.340057 26775 consensus_queue.cc:260] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.340319 26778 tablet_bootstrap.cc:492] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888: Bootstrap starting.
I20240501 20:11:44.340685 26775 ts_tablet_manager.cc:1398] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:44.341063 26775 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0: Bootstrap starting.
I20240501 20:11:44.341603 26774 raft_consensus.cc:348] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.341840 26774 raft_consensus.cc:374] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:44.341953 26774 raft_consensus.cc:729] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Initialized, Role: FOLLOWER
I20240501 20:11:44.342092 26778 tablet_bootstrap.cc:654] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:44.342265 26774 consensus_queue.cc:260] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.343354 26774 ts_tablet_manager.cc:1398] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 20:11:44.343570 26775 tablet_bootstrap.cc:654] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:44.343708 26774 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9: Bootstrap starting.
I20240501 20:11:44.345247 26774 tablet_bootstrap.cc:654] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:44.351369 26775 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0: No bootstrap required, opened a new log
I20240501 20:11:44.351518 26775 ts_tablet_manager.cc:1367] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.003s
I20240501 20:11:44.351753 26778 tablet_bootstrap.cc:492] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888: No bootstrap required, opened a new log
I20240501 20:11:44.351889 26778 ts_tablet_manager.cc:1367] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888: Time spent bootstrapping tablet: real 0.012s	user 0.011s	sys 0.000s
I20240501 20:11:44.352327 26775 raft_consensus.cc:348] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.352458 26778 raft_consensus.cc:348] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.352572 26775 raft_consensus.cc:374] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:44.352643 26778 raft_consensus.cc:374] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:44.352661 26775 raft_consensus.cc:729] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:44.352749 26778 raft_consensus.cc:729] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Initialized, Role: FOLLOWER
I20240501 20:11:44.352916 26775 consensus_queue.cc:260] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.352985 26778 consensus_queue.cc:260] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.353462 26778 ts_tablet_manager.cc:1398] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240501 20:11:44.353487 26775 ts_tablet_manager.cc:1398] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:11:44.353706 26774 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9: No bootstrap required, opened a new log
I20240501 20:11:44.353843 26774 ts_tablet_manager.cc:1367] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.008s
I20240501 20:11:44.353929 26775 tablet_bootstrap.cc:492] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0: Bootstrap starting.
I20240501 20:11:44.354565 26774 raft_consensus.cc:348] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.354766 26774 raft_consensus.cc:374] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:44.354861 26774 raft_consensus.cc:729] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Initialized, Role: FOLLOWER
I20240501 20:11:44.355064 26774 consensus_queue.cc:260] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.355580 26774 ts_tablet_manager.cc:1398] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:11:44.355589 26775 tablet_bootstrap.cc:654] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:44.355875 26778 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888: Bootstrap starting.
I20240501 20:11:44.357383 26778 tablet_bootstrap.cc:654] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:44.361349 26775 tablet_bootstrap.cc:492] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0: No bootstrap required, opened a new log
I20240501 20:11:44.361477 26775 ts_tablet_manager.cc:1367] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240501 20:11:44.361554 26778 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888: No bootstrap required, opened a new log
I20240501 20:11:44.361725 26778 ts_tablet_manager.cc:1367] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.003s
I20240501 20:11:44.362056 26775 raft_consensus.cc:348] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.362248 26775 raft_consensus.cc:374] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:44.362363 26775 raft_consensus.cc:729] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:44.362419 26778 raft_consensus.cc:348] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.362612 26778 raft_consensus.cc:374] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:44.362694 26778 raft_consensus.cc:729] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Initialized, Role: FOLLOWER
I20240501 20:11:44.362583 26775 consensus_queue.cc:260] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.363068 26775 ts_tablet_manager.cc:1398] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240501 20:11:44.362967 26778 consensus_queue.cc:260] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.363488 26778 ts_tablet_manager.cc:1398] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:11:44.364022 26778 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888: Bootstrap starting.
I20240501 20:11:44.365444 26778 tablet_bootstrap.cc:654] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:44.367424 26778 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888: No bootstrap required, opened a new log
I20240501 20:11:44.367568 26778 ts_tablet_manager.cc:1367] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:11:44.368165 26778 raft_consensus.cc:348] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.368294 26778 raft_consensus.cc:374] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:44.368355 26778 raft_consensus.cc:729] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Initialized, Role: FOLLOWER
I20240501 20:11:44.368532 26778 consensus_queue.cc:260] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.368909 26778 ts_tablet_manager.cc:1398] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:44.490715 26754 raft_consensus.cc:482] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:44.490895 26754 raft_consensus.cc:504] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.491652 26754 leader_election.cc:288] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:44.492077 26077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d4fc9946d44e548a0bac91834b8afd" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" is_pre_election: true
I20240501 20:11:44.492120 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d4fc9946d44e548a0bac91834b8afd" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" is_pre_election: true
I20240501 20:11:44.492326 26345 raft_consensus.cc:2454] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:44.492311 26077 raft_consensus.cc:2454] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:44.492725 26147 leader_election.cc:302] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, 41abd2deb0d7412bbb66b1aa9851d888; no voters: 
I20240501 20:11:44.492974 26754 raft_consensus.cc:2781] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:44.493088 26754 raft_consensus.cc:482] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:44.493155 26754 raft_consensus.cc:3037] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:44.494927 26767 raft_consensus.cc:482] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:44.495092 26754 raft_consensus.cc:504] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.495074 26767 raft_consensus.cc:504] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.495829 26767 leader_election.cc:288] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:44.495854 26754 leader_election.cc:288] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Requested vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:44.496228 26078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57621c56ad05419aaafd7036ced2b691" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" is_pre_election: true
I20240501 20:11:44.496284 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57621c56ad05419aaafd7036ced2b691" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41abd2deb0d7412bbb66b1aa9851d888" is_pre_election: true
I20240501 20:11:44.496258 26077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d4fc9946d44e548a0bac91834b8afd" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
I20240501 20:11:44.496454 26213 raft_consensus.cc:2454] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 0.
I20240501 20:11:44.496490 26078 raft_consensus.cc:2454] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 0.
I20240501 20:11:44.496443 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d4fc9946d44e548a0bac91834b8afd" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
I20240501 20:11:44.496495 26077 raft_consensus.cc:3037] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:44.496677 26345 raft_consensus.cc:3037] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:44.498484 26281 leader_election.cc:302] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:44.498723 26767 raft_consensus.cc:2781] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:44.498836 26767 raft_consensus.cc:482] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:44.498924 26767 raft_consensus.cc:3037] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:44.499293 26077 raft_consensus.cc:2454] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:44.499296 26345 raft_consensus.cc:2454] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:44.499663 26147 leader_election.cc:302] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, 41abd2deb0d7412bbb66b1aa9851d888; no voters: 
I20240501 20:11:44.500011 26754 raft_consensus.cc:2781] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:44.500174 26754 raft_consensus.cc:686] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Becoming Leader. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Running, Role: LEADER
I20240501 20:11:44.500586 26754 consensus_queue.cc:237] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.501000 26767 raft_consensus.cc:504] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.501539 26767 leader_election.cc:288] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 election: Requested vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:44.502103 26077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57621c56ad05419aaafd7036ced2b691" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
I20240501 20:11:44.502156 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57621c56ad05419aaafd7036ced2b691" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41abd2deb0d7412bbb66b1aa9851d888"
I20240501 20:11:44.502365 26077 raft_consensus.cc:3037] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:44.502470 26213 raft_consensus.cc:3037] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:44.503355 25931 catalog_manager.cc:5484] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:44.505030 26077 raft_consensus.cc:2454] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 1.
I20240501 20:11:44.505101 26213 raft_consensus.cc:2454] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 1.
I20240501 20:11:44.505466 26281 leader_election.cc:302] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, 41abd2deb0d7412bbb66b1aa9851d888; no voters: 
I20240501 20:11:44.505748 26767 raft_consensus.cc:2781] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:44.505920 26767 raft_consensus.cc:686] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Running, Role: LEADER
I20240501 20:11:44.506227 26767 consensus_queue.cc:237] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.508464 25931 catalog_manager.cc:5484] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:44.673696 26754 raft_consensus.cc:482] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:44.673868 26754 raft_consensus.cc:504] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.674468 26754 leader_election.cc:288] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:44.674880 26077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f145d7f939459fbc804b732189db3f" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" is_pre_election: true
I20240501 20:11:44.674944 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f145d7f939459fbc804b732189db3f" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" is_pre_election: true
I20240501 20:11:44.675074 26077 raft_consensus.cc:2454] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:44.675137 26345 raft_consensus.cc:2454] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:44.675457 26148 leader_election.cc:302] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41abd2deb0d7412bbb66b1aa9851d888, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:44.675736 26754 raft_consensus.cc:2781] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:44.675832 26754 raft_consensus.cc:482] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:44.675925 26754 raft_consensus.cc:3037] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:44.677716 26754 raft_consensus.cc:504] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.678375 26754 leader_election.cc:288] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Requested vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:44.678661 26077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f145d7f939459fbc804b732189db3f" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
I20240501 20:11:44.678830 26077 raft_consensus.cc:3037] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:44.678970 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f145d7f939459fbc804b732189db3f" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
I20240501 20:11:44.679200 26345 raft_consensus.cc:3037] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:44.680706 26077 raft_consensus.cc:2454] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:44.681032 26148 leader_election.cc:302] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41abd2deb0d7412bbb66b1aa9851d888, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:44.681032 26345 raft_consensus.cc:2454] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:44.681262 26754 raft_consensus.cc:2781] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:44.681407 26754 raft_consensus.cc:686] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Becoming Leader. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Running, Role: LEADER
I20240501 20:11:44.681710 26754 consensus_queue.cc:237] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:44.684010 25931 catalog_manager.cc:5484] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:45.010962 26789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:45.011236 26789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:45.018338 26789 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:45.033056 26767 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:45.049445 26767 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:45.057044 26754 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:45.058877 26754 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:45.065584 26754 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 20:11:45.082715 26768 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 192.855 ms
  time per row: 0.0964275 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 20:11:46.233223 25924 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41972:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 20:11:46.233933 25924 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:11:46.242465 26323 tablet_service.cc:1467] Processing CreateTablet for tablet 3f369d7f22f34caaa70a2e167165a0d3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:46.242945 26324 tablet_service.cc:1467] Processing CreateTablet for tablet b8625da0d7164bf0b18052e5be5013df (DEFAULT_TABLE table=rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:46.243602 26056 tablet_service.cc:1467] Processing CreateTablet for tablet 3f369d7f22f34caaa70a2e167165a0d3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:46.243799 26325 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4ef284f1844f80aed689e14187638d (DEFAULT_TABLE table=rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:46.243808 26057 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4ef284f1844f80aed689e14187638d (DEFAULT_TABLE table=rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:46.244063 26056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f369d7f22f34caaa70a2e167165a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.244064 26191 tablet_service.cc:1467] Processing CreateTablet for tablet 3f369d7f22f34caaa70a2e167165a0d3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:46.244309 26057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4ef284f1844f80aed689e14187638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.244439 26191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f369d7f22f34caaa70a2e167165a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.244450 26058 tablet_service.cc:1467] Processing CreateTablet for tablet b8625da0d7164bf0b18052e5be5013df (DEFAULT_TABLE table=rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:46.244796 26058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8625da0d7164bf0b18052e5be5013df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.248278 26192 tablet_service.cc:1467] Processing CreateTablet for tablet b8625da0d7164bf0b18052e5be5013df (DEFAULT_TABLE table=rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:46.248592 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8625da0d7164bf0b18052e5be5013df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.242938 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f369d7f22f34caaa70a2e167165a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.250555 26324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8625da0d7164bf0b18052e5be5013df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.250564 26193 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4ef284f1844f80aed689e14187638d (DEFAULT_TABLE table=rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:46.250918 26193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4ef284f1844f80aed689e14187638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.253835 26325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4ef284f1844f80aed689e14187638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.260486 26857 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9: Bootstrap starting.
I20240501 20:11:46.262176 26857 tablet_bootstrap.cc:654] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.266723 26859 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888: Bootstrap starting.
I20240501 20:11:46.266727 26856 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0: Bootstrap starting.
I20240501 20:11:46.268162 26856 tablet_bootstrap.cc:654] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.269093 26859 tablet_bootstrap.cc:654] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.269896 26857 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9: No bootstrap required, opened a new log
I20240501 20:11:46.270082 26857 ts_tablet_manager.cc:1367] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.005s
I20240501 20:11:46.270155 26856 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0: No bootstrap required, opened a new log
I20240501 20:11:46.270385 26856 ts_tablet_manager.cc:1367] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240501 20:11:46.270870 26859 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888: No bootstrap required, opened a new log
I20240501 20:11:46.270781 26857 raft_consensus.cc:348] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.270960 26857 raft_consensus.cc:374] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.271000 26859 ts_tablet_manager.cc:1367] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:11:46.271100 26857 raft_consensus.cc:729] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.271040 26856 raft_consensus.cc:348] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.271355 26856 raft_consensus.cc:374] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.271462 26856 raft_consensus.cc:729] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.271386 26857 consensus_queue.cc:260] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.271689 26856 consensus_queue.cc:260] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.271991 26857 ts_tablet_manager.cc:1398] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 20:11:46.272238 26857 tablet_bootstrap.cc:492] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9: Bootstrap starting.
I20240501 20:11:46.272226 26856 ts_tablet_manager.cc:1398] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:11:46.272488 26856 tablet_bootstrap.cc:492] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0: Bootstrap starting.
I20240501 20:11:46.273691 26857 tablet_bootstrap.cc:654] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.273975 26856 tablet_bootstrap.cc:654] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.278349 26859 raft_consensus.cc:348] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.278565 26859 raft_consensus.cc:374] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.278693 26859 raft_consensus.cc:729] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.278913 26859 consensus_queue.cc:260] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.280633 26859 ts_tablet_manager.cc:1398] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240501 20:11:46.280948 26859 tablet_bootstrap.cc:492] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888: Bootstrap starting.
I20240501 20:11:46.282184 26856 tablet_bootstrap.cc:492] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0: No bootstrap required, opened a new log
I20240501 20:11:46.282356 26856 ts_tablet_manager.cc:1367] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.001s
I20240501 20:11:46.282492 26859 tablet_bootstrap.cc:654] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.283092 26856 raft_consensus.cc:348] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.283303 26856 raft_consensus.cc:374] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.283370 26856 raft_consensus.cc:729] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.283666 26856 consensus_queue.cc:260] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.284206 26856 ts_tablet_manager.cc:1398] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:11:46.284519 26856 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0: Bootstrap starting.
I20240501 20:11:46.285192 26859 tablet_bootstrap.cc:492] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888: No bootstrap required, opened a new log
I20240501 20:11:46.285988 26856 tablet_bootstrap.cc:654] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.285334 26859 ts_tablet_manager.cc:1367] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 20:11:46.287173 26859 raft_consensus.cc:348] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.287410 26859 raft_consensus.cc:374] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.287494 26859 raft_consensus.cc:729] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.287829 26859 consensus_queue.cc:260] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.288440 26859 ts_tablet_manager.cc:1398] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:11:46.288769 26859 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888: Bootstrap starting.
I20240501 20:11:46.290338 26859 tablet_bootstrap.cc:654] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.292116 26857 tablet_bootstrap.cc:492] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9: No bootstrap required, opened a new log
I20240501 20:11:46.292330 26857 ts_tablet_manager.cc:1367] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.012s
I20240501 20:11:46.293033 26857 raft_consensus.cc:348] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.293282 26857 raft_consensus.cc:374] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.293395 26857 raft_consensus.cc:729] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.293787 26856 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0: No bootstrap required, opened a new log
I20240501 20:11:46.293673 26857 consensus_queue.cc:260] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.293905 26856 ts_tablet_manager.cc:1367] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.004s
I20240501 20:11:46.294224 26857 ts_tablet_manager.cc:1398] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 20:11:46.294560 26857 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9: Bootstrap starting.
I20240501 20:11:46.295086 26856 raft_consensus.cc:348] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.295308 26856 raft_consensus.cc:374] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.295415 26856 raft_consensus.cc:729] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.295640 26856 consensus_queue.cc:260] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.296018 26857 tablet_bootstrap.cc:654] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.296090 26856 ts_tablet_manager.cc:1398] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:11:46.297883 26857 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9: No bootstrap required, opened a new log
I20240501 20:11:46.298053 26857 ts_tablet_manager.cc:1367] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 20:11:46.298782 26857 raft_consensus.cc:348] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.299010 26857 raft_consensus.cc:374] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.299059 26859 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888: No bootstrap required, opened a new log
I20240501 20:11:46.299153 26857 raft_consensus.cc:729] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.299188 26859 ts_tablet_manager.cc:1367] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.000s
I20240501 20:11:46.299387 26857 consensus_queue.cc:260] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.299829 26859 raft_consensus.cc:348] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.299952 26857 ts_tablet_manager.cc:1398] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:11:46.300069 26859 raft_consensus.cc:374] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.300165 26859 raft_consensus.cc:729] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.300354 26859 consensus_queue.cc:260] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.300716 26859 ts_tablet_manager.cc:1398] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240501 20:11:46.327586 26800 raft_consensus.cc:482] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:46.327710 26800 raft_consensus.cc:504] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.328244 26800 leader_election.cc:288] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:46.328642 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4ef284f1844f80aed689e14187638d" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41abd2deb0d7412bbb66b1aa9851d888" is_pre_election: true
I20240501 20:11:46.328655 26076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4ef284f1844f80aed689e14187638d" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" is_pre_election: true
I20240501 20:11:46.328833 26076 raft_consensus.cc:2454] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 0.
I20240501 20:11:46.328856 26213 raft_consensus.cc:2454] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 0.
I20240501 20:11:46.329226 26281 leader_election.cc:302] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:46.329458 26800 raft_consensus.cc:2781] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:46.329571 26800 raft_consensus.cc:482] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:46.329655 26800 raft_consensus.cc:3037] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.331478 26800 raft_consensus.cc:504] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.331984 26800 leader_election.cc:288] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 election: Requested vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:46.332314 26076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4ef284f1844f80aed689e14187638d" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
I20240501 20:11:46.332497 26076 raft_consensus.cc:3037] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.332450 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4ef284f1844f80aed689e14187638d" candidate_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41abd2deb0d7412bbb66b1aa9851d888"
I20240501 20:11:46.332629 26212 raft_consensus.cc:3037] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.334390 26076 raft_consensus.cc:2454] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 1.
I20240501 20:11:46.334403 26212 raft_consensus.cc:2454] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c992b98bb0d4f1b92d058c882fb68b9 in term 1.
I20240501 20:11:46.334705 26281 leader_election.cc:302] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:46.334954 26800 raft_consensus.cc:2781] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:46.335255 26800 raft_consensus.cc:686] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1c992b98bb0d4f1b92d058c882fb68b9, State: Running, Role: LEADER
I20240501 20:11:46.335506 26800 consensus_queue.cc:237] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.337815 25931 catalog_manager.cc:5484] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195). New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:46.352348 26837 raft_consensus.cc:482] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:46.352490 26837 raft_consensus.cc:504] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.353019 26837 leader_election.cc:288] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:46.353384 26076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f369d7f22f34caaa70a2e167165a0d3" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" is_pre_election: true
I20240501 20:11:46.353399 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f369d7f22f34caaa70a2e167165a0d3" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" is_pre_election: true
I20240501 20:11:46.353545 26344 raft_consensus.cc:2454] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:46.353569 26076 raft_consensus.cc:2454] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:46.353946 26147 leader_election.cc:302] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, 41abd2deb0d7412bbb66b1aa9851d888; no voters: 
I20240501 20:11:46.354219 26837 raft_consensus.cc:2781] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:46.354342 26837 raft_consensus.cc:482] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:46.354430 26837 raft_consensus.cc:3037] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.356240 26837 raft_consensus.cc:504] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.356812 26837 leader_election.cc:288] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Requested vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:46.357192 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f369d7f22f34caaa70a2e167165a0d3" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
I20240501 20:11:46.357232 26076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f369d7f22f34caaa70a2e167165a0d3" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
I20240501 20:11:46.357336 26344 raft_consensus.cc:3037] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.357391 26076 raft_consensus.cc:3037] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.359326 26344 raft_consensus.cc:2454] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:46.359318 26076 raft_consensus.cc:2454] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:46.359659 26147 leader_election.cc:302] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, 41abd2deb0d7412bbb66b1aa9851d888, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:46.359995 26870 raft_consensus.cc:2781] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:46.360170 26870 raft_consensus.cc:686] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Becoming Leader. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Running, Role: LEADER
I20240501 20:11:46.360469 26870 consensus_queue.cc:237] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.362771 25931 catalog_manager.cc:5484] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:46.380394 26870 raft_consensus.cc:482] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:46.380529 26870 raft_consensus.cc:504] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.380980 26870 leader_election.cc:288] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:46.381374 26076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8625da0d7164bf0b18052e5be5013df" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" is_pre_election: true
I20240501 20:11:46.381393 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8625da0d7164bf0b18052e5be5013df" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" is_pre_election: true
I20240501 20:11:46.381549 26076 raft_consensus.cc:2454] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:46.381558 26344 raft_consensus.cc:2454] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 0.
I20240501 20:11:46.381894 26147 leader_election.cc:302] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, 41abd2deb0d7412bbb66b1aa9851d888; no voters: 
I20240501 20:11:46.382166 26870 raft_consensus.cc:2781] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:46.382261 26870 raft_consensus.cc:482] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:46.382381 26870 raft_consensus.cc:3037] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.384167 26870 raft_consensus.cc:504] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.384676 26870 leader_election.cc:288] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Requested vote from peers e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:46.385043 26076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8625da0d7164bf0b18052e5be5013df" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
I20240501 20:11:46.385087 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8625da0d7164bf0b18052e5be5013df" candidate_uuid: "41abd2deb0d7412bbb66b1aa9851d888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
I20240501 20:11:46.385229 26076 raft_consensus.cc:3037] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.385259 26344 raft_consensus.cc:3037] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.387828 26344 raft_consensus.cc:2454] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:46.387846 26076 raft_consensus.cc:2454] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41abd2deb0d7412bbb66b1aa9851d888 in term 1.
I20240501 20:11:46.388190 26147 leader_election.cc:302] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, 41abd2deb0d7412bbb66b1aa9851d888, e9a0d830efeb4d43b13221b36d8e77f0; no voters: 
I20240501 20:11:46.388453 26870 raft_consensus.cc:2781] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:46.388605 26870 raft_consensus.cc:686] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Becoming Leader. State: Replica: 41abd2deb0d7412bbb66b1aa9851d888, State: Running, Role: LEADER
I20240501 20:11:46.388833 26870 consensus_queue.cc:237] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:46.390936 25931 catalog_manager.cc:5484] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194). New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:46.729717 26873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:46.729992 26873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:46.737586 26873 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:46.743268 26800 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:46.747979 26800 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 20:11:46.775592 26870 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:46.780604 26895 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:46.842013 26895 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:46.848511 26837 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 244.771 ms
  time per row: 0.122385 ms
I20240501 20:11:47.133558  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.197:46545
--local_ip_for_outbound_sockets=127.8.68.197
--tserver_master_addrs=127.8.68.254:36731
--webserver_port=38925
--webserver_interface=127.8.68.197
--builtin_ntp_servers=127.8.68.212:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:47.454531 26937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:47.454820 26937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:47.455013 26937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:47.462399 26937 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:47.462497 26937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:47.462641 26937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.197
I20240501 20:11:47.471663 26937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.197:46545
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.68.197
--webserver_port=38925
--tserver_master_addrs=127.8.68.254:36731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.197
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:47.472716 26937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:47.474244 26937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:47.485316 26943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:47.485783 26942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:47.486308 26945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:47.486817 26937 server_base.cc:1034] running on GCE node
I20240501 20:11:47.487289 26937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:47.488092 26937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:47.489284 26937 hybrid_clock.cc:648] HybridClock initialized: now 1714594307489241 us; error 36 us; skew 500 ppm
I20240501 20:11:47.489593 26937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:47.491676 26937 webserver.cc:460] Webserver started at http://127.8.68.197:38925/ using document root <none> and password file <none>
I20240501 20:11:47.492303 26937 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:47.492417 26937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:47.497224 26937 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 20:11:47.499644 26951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:47.500753 26937 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I20240501 20:11:47.500885 26937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f79d4befcb2c4d6cb0e1b79379164c25"
format_stamp: "Formatted at 2024-05-01 20:11:39 on dist-test-slave-08p9"
I20240501 20:11:47.501608 26937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:47.538960 26937 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:47.539610 26937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:47.539824 26937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:47.540654 26937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:47.542018 26937 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:47.542091 26937 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:47.542232 26937 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:47.542313 26937 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:47.570751 26937 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.197:46545
I20240501 20:11:47.570817 27064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.197:46545 every 8 connection(s)
I20240501 20:11:47.571975 26937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:11:47.579754  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 26937
I20240501 20:11:47.580540  8467 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
/tmp/dist-test-taskH_e3f3/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.68.196:34631
--local_ip_for_outbound_sockets=127.8.68.196
--tserver_master_addrs=127.8.68.254:36731
--webserver_port=45127
--webserver_interface=127.8.68.196
--builtin_ntp_servers=127.8.68.212:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:47.590801 27065 heartbeater.cc:347] Connected to a master server at 127.8.68.254:36731
I20240501 20:11:47.591058 27065 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:47.591796 27065 heartbeater.cc:511] Master 127.8.68.254:36731 requested a full tablet report, sending...
I20240501 20:11:47.593405 25930 ts_manager.cc:194] Re-registered known tserver with Master: f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545)
I20240501 20:11:47.594266 25930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.197:57361
W20240501 20:11:47.893457 27069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:47.893733 27069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:47.893864 27069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:47.901358 27069 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:47.901477 27069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:47.901610 27069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.68.196
I20240501 20:11:47.910027 27069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.68.212:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.68.196:34631
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.68.196
--webserver_port=45127
--tserver_master_addrs=127.8.68.254:36731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.68.196
--log_dir=/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:47.911091 27069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:47.912613 27069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:47.923413 27074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:47.923929 27075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:47.924297 27077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:47.925056 27069 server_base.cc:1034] running on GCE node
I20240501 20:11:47.925565 27069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:47.926391 27069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:47.927577 27069 hybrid_clock.cc:648] HybridClock initialized: now 1714594307927532 us; error 38 us; skew 500 ppm
I20240501 20:11:47.927892 27069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:47.930052 27069 webserver.cc:460] Webserver started at http://127.8.68.196:45127/ using document root <none> and password file <none>
I20240501 20:11:47.930748 27069 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:47.930872 27069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:47.935485 27069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.002s
I20240501 20:11:47.937819 27083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:47.939038 27069 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 20:11:47.939184 27069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0afaccbf1fb84dd7b530d6632c4cd948"
format_stamp: "Formatted at 2024-05-01 20:11:38 on dist-test-slave-08p9"
I20240501 20:11:47.939901 27069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:47.951560 27069 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 20:11:47.952183 27069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:47.952410 27069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:47.953178 27069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:47.954515 27069 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:47.954587 27069 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:47.954699 27069 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:47.954748 27069 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 20:11:47.983659 27069 rpc_server.cc:307] RPC server started. Bound to: 127.8.68.196:34631
I20240501 20:11:47.983723 27195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.68.196:34631 every 8 connection(s)
I20240501 20:11:47.985143 27069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:47.991376  8467 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu as pid 27069
I20240501 20:11:48.001305 27196 heartbeater.cc:347] Connected to a master server at 127.8.68.254:36731
I20240501 20:11:48.001579 27196 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:48.002455 27196 heartbeater.cc:511] Master 127.8.68.254:36731 requested a full tablet report, sending...
I20240501 20:11:48.004019 25930 ts_manager.cc:194] Re-registered known tserver with Master: 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631)
I20240501 20:11:48.004982 25930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.68.196:50271
I20240501 20:11:48.440915 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.467626 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:48.468684 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:48.480648 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.481037 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:48.596786 27065 heartbeater.cc:503] Master 127.8.68.254:36731 was elected leader, sending a full tablet report...
I20240501 20:11:48.642642 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.643796 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:48.650346 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:48.660612 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.677178 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:48.834892 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.840044 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:48.844190 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:48.848727 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:48.867539 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:48.997639 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.007266 27196 heartbeater.cc:503] Master 127.8.68.254:36731 was elected leader, sending a full tablet report...
I20240501 20:11:49.016481 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:49.019754 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:49.044041 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:49.046435 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:49.106966 26212 consensus_queue.cc:237] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:49.109849 26344 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:49.109948 26078 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:49.110401 26919 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.110803 27340 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.113426 26919 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.118818 26078 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.119748 26344 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.128167 25932 catalog_manager.cc:5484] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 reported cstate change: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
W20240501 20:11:49.139449 26146 consensus_peers.cc:485] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 -> Peer 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631): Couldn't send request to peer 0afaccbf1fb84dd7b530d6632c4cd948. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f369d7f22f34caaa70a2e167165a0d3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:49.141283 26344 consensus_queue.cc:237] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:49.143486 26078 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:49.143936 26800 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.144395 26212 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:49.144893 26800 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.146698 26800 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:49.150595 26212 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:49.150775 26078 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
W20240501 20:11:49.151257 26280 consensus_peers.cc:485] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 -> Peer f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545): Couldn't send request to peer f79d4befcb2c4d6cb0e1b79379164c25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57621c56ad05419aaafd7036ced2b691. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:49.155544 25926 catalog_manager.cc:5484] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:49.167871 26212 consensus_queue.cc:237] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:49.169715 26078 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:49.169857 26344 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:49.170131 26146 consensus_peers.cc:485] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 -> Peer 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631): Couldn't send request to peer 0afaccbf1fb84dd7b530d6632c4cd948. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd17acc8002948999a5c607ccb9bb189. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:49.170308 27340 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.170676 27345 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.173274 27341 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.173913 26344 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.173883 26078 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.178514 25932 catalog_manager.cc:5484] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.190704 26078 consensus_queue.cc:237] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:49.193944 26212 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:49.193910 26344 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:49.194500 27376 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.194864 26731 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.198035 26731 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:49.198577 26212 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:49.204309 25930 catalog_manager.cc:5484] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
W20240501 20:11:49.211181 26012 consensus_peers.cc:485] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 -> Peer f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545): Couldn't send request to peer f79d4befcb2c4d6cb0e1b79379164c25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f35f4b0292e045949cae7b7f4120b67c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:49.215721 26344 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:49.230406 26212 consensus_queue.cc:237] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:49.232456 26078 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:49.232518 26344 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:49.233000 26146 consensus_peers.cc:485] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 -> Peer 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631): Couldn't send request to peer 0afaccbf1fb84dd7b530d6632c4cd948. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97f145d7f939459fbc804b732189db3f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:49.233053 27341 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.233386 26919 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.235944 27341 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.236507 26344 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.236701 26078 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.240224 25930 catalog_manager.cc:5484] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index -1 to 4, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:49.252028 26344 consensus_queue.cc:237] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:49.254019 26212 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:49.254019 26078 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:49.254186 26280 consensus_peers.cc:485] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 -> Peer f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545): Couldn't send request to peer f79d4befcb2c4d6cb0e1b79379164c25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c4ef284f1844f80aed689e14187638d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:49.254626 27361 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.254879 27360 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:49.257436 27356 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:49.258023 26078 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:49.258111 26212 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:49.261615 25926 catalog_manager.cc:5484] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index -1 to 4, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:49.577842 27459 ts_tablet_manager.cc:902] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: Initiating tablet copy from peer 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:49.579871 27459 tablet_copy_client.cc:308] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:35909
I20240501 20:11:49.586932 26233 tablet_copy_service.cc:140] P 41abd2deb0d7412bbb66b1aa9851d888: Received BeginTabletCopySession request for tablet cd17acc8002948999a5c607ccb9bb189 from peer 0afaccbf1fb84dd7b530d6632c4cd948 ({username='slave'} at 127.8.68.196:55851)
I20240501 20:11:49.587141 26233 tablet_copy_service.cc:161] P 41abd2deb0d7412bbb66b1aa9851d888: Beginning new tablet copy session on tablet cd17acc8002948999a5c607ccb9bb189 from peer 0afaccbf1fb84dd7b530d6632c4cd948 at {username='slave'} at 127.8.68.196:55851: session id = 0afaccbf1fb84dd7b530d6632c4cd948-cd17acc8002948999a5c607ccb9bb189
I20240501 20:11:49.589650 26233 tablet_copy_source_session.cc:205] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:49.594646 27459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd17acc8002948999a5c607ccb9bb189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:49.605163 27459 tablet_copy_client.cc:783] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:49.605446 27459 tablet_copy_client.cc:647] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:49.607915 27459 tablet_copy_client.cc:523] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:49.610725 27459 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap starting.
I20240501 20:11:49.618908 27459 log.cc:824] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:49.630807 27466 ts_tablet_manager.cc:902] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: Initiating tablet copy from peer 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:49.630870 27467 ts_tablet_manager.cc:902] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: Initiating tablet copy from peer 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:49.632905 27467 tablet_copy_client.cc:308] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:37745
I20240501 20:11:49.634891 27466 tablet_copy_client.cc:308] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:37745
I20240501 20:11:49.640830 26365 tablet_copy_service.cc:140] P 1c992b98bb0d4f1b92d058c882fb68b9: Received BeginTabletCopySession request for tablet 57621c56ad05419aaafd7036ced2b691 from peer f79d4befcb2c4d6cb0e1b79379164c25 ({username='slave'} at 127.8.68.197:52555)
I20240501 20:11:49.640879 26364 tablet_copy_service.cc:140] P 1c992b98bb0d4f1b92d058c882fb68b9: Received BeginTabletCopySession request for tablet 6c4ef284f1844f80aed689e14187638d from peer f79d4befcb2c4d6cb0e1b79379164c25 ({username='slave'} at 127.8.68.197:52555)
I20240501 20:11:49.641124 26365 tablet_copy_service.cc:161] P 1c992b98bb0d4f1b92d058c882fb68b9: Beginning new tablet copy session on tablet 57621c56ad05419aaafd7036ced2b691 from peer f79d4befcb2c4d6cb0e1b79379164c25 at {username='slave'} at 127.8.68.197:52555: session id = f79d4befcb2c4d6cb0e1b79379164c25-57621c56ad05419aaafd7036ced2b691
I20240501 20:11:49.641603 26364 tablet_copy_service.cc:161] P 1c992b98bb0d4f1b92d058c882fb68b9: Beginning new tablet copy session on tablet 6c4ef284f1844f80aed689e14187638d from peer f79d4befcb2c4d6cb0e1b79379164c25 at {username='slave'} at 127.8.68.197:52555: session id = f79d4befcb2c4d6cb0e1b79379164c25-6c4ef284f1844f80aed689e14187638d
I20240501 20:11:49.645251 26364 tablet_copy_source_session.cc:205] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:49.645602 27477 ts_tablet_manager.cc:902] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: Initiating tablet copy from peer 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:49.647202 27477 tablet_copy_client.cc:308] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:35909
I20240501 20:11:49.647612 26365 tablet_copy_source_session.cc:205] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:49.647810 26233 tablet_copy_service.cc:140] P 41abd2deb0d7412bbb66b1aa9851d888: Received BeginTabletCopySession request for tablet 97f145d7f939459fbc804b732189db3f from peer 0afaccbf1fb84dd7b530d6632c4cd948 ({username='slave'} at 127.8.68.196:55851)
I20240501 20:11:49.647974 26233 tablet_copy_service.cc:161] P 41abd2deb0d7412bbb66b1aa9851d888: Beginning new tablet copy session on tablet 97f145d7f939459fbc804b732189db3f from peer 0afaccbf1fb84dd7b530d6632c4cd948 at {username='slave'} at 127.8.68.196:55851: session id = 0afaccbf1fb84dd7b530d6632c4cd948-97f145d7f939459fbc804b732189db3f
I20240501 20:11:49.650210 27467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57621c56ad05419aaafd7036ced2b691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:49.650585 26233 tablet_copy_source_session.cc:205] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:49.650687 27466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4ef284f1844f80aed689e14187638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:49.651698 27477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f145d7f939459fbc804b732189db3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:49.657289 27477 tablet_copy_client.cc:783] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:49.657737 27477 tablet_copy_client.cc:647] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:49.659368 27477 tablet_copy_client.cc:523] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:49.660388 27459 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:49.660990 27459 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap complete.
I20240501 20:11:49.661485 27459 ts_tablet_manager.cc:1367] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent bootstrapping tablet: real 0.051s	user 0.048s	sys 0.000s
I20240501 20:11:49.662946 27477 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap starting.
I20240501 20:11:49.663895 27466 tablet_copy_client.cc:783] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:49.664196 27467 tablet_copy_client.cc:783] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:49.664240 27466 tablet_copy_client.cc:647] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:49.664489 27467 tablet_copy_client.cc:647] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:49.666785 27467 tablet_copy_client.cc:523] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:49.666955 27466 tablet_copy_client.cc:523] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:49.667899 27459 raft_consensus.cc:348] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:49.668442 27459 raft_consensus.cc:729] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0afaccbf1fb84dd7b530d6632c4cd948, State: Initialized, Role: LEARNER
I20240501 20:11:49.669018 27459 consensus_queue.cc:260] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:49.670997 27467 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap starting.
I20240501 20:11:49.671028 27466 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap starting.
I20240501 20:11:49.676997 27459 ts_tablet_manager.cc:1398] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.007s
I20240501 20:11:49.678700 26233 tablet_copy_service.cc:342] P 41abd2deb0d7412bbb66b1aa9851d888: Request end of tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-cd17acc8002948999a5c607ccb9bb189 received from {username='slave'} at 127.8.68.196:55851
I20240501 20:11:49.678851 26233 tablet_copy_service.cc:433] P 41abd2deb0d7412bbb66b1aa9851d888: ending tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-cd17acc8002948999a5c607ccb9bb189 on tablet cd17acc8002948999a5c607ccb9bb189 with peer 0afaccbf1fb84dd7b530d6632c4cd948
I20240501 20:11:49.683813 27467 log.cc:824] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:49.697685 27459 ts_tablet_manager.cc:902] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: Initiating tablet copy from peer 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:49.716878 27477 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:49.717599 27477 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap complete.
I20240501 20:11:49.717938 27477 ts_tablet_manager.cc:1367] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent bootstrapping tablet: real 0.055s	user 0.038s	sys 0.014s
I20240501 20:11:49.719699 27477 raft_consensus.cc:348] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:49.720034 27477 raft_consensus.cc:729] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0afaccbf1fb84dd7b530d6632c4cd948, State: Initialized, Role: LEARNER
I20240501 20:11:49.720449 27477 consensus_queue.cc:260] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:49.721238 27477 ts_tablet_manager.cc:1398] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 20:11:49.721966 26233 tablet_copy_service.cc:342] P 41abd2deb0d7412bbb66b1aa9851d888: Request end of tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-97f145d7f939459fbc804b732189db3f received from {username='slave'} at 127.8.68.196:55851
I20240501 20:11:49.722088 26233 tablet_copy_service.cc:433] P 41abd2deb0d7412bbb66b1aa9851d888: ending tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-97f145d7f939459fbc804b732189db3f on tablet 97f145d7f939459fbc804b732189db3f with peer 0afaccbf1fb84dd7b530d6632c4cd948
I20240501 20:11:49.726764 27459 tablet_copy_client.cc:308] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:35909
I20240501 20:11:49.727440 27467 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:49.727411 26233 tablet_copy_service.cc:140] P 41abd2deb0d7412bbb66b1aa9851d888: Received BeginTabletCopySession request for tablet 3f369d7f22f34caaa70a2e167165a0d3 from peer 0afaccbf1fb84dd7b530d6632c4cd948 ({username='slave'} at 127.8.68.196:55851)
I20240501 20:11:49.727546 26233 tablet_copy_service.cc:161] P 41abd2deb0d7412bbb66b1aa9851d888: Beginning new tablet copy session on tablet 3f369d7f22f34caaa70a2e167165a0d3 from peer 0afaccbf1fb84dd7b530d6632c4cd948 at {username='slave'} at 127.8.68.196:55851: session id = 0afaccbf1fb84dd7b530d6632c4cd948-3f369d7f22f34caaa70a2e167165a0d3
I20240501 20:11:49.728058 27467 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap complete.
I20240501 20:11:49.728513 27467 ts_tablet_manager.cc:1367] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: Time spent bootstrapping tablet: real 0.058s	user 0.049s	sys 0.004s
I20240501 20:11:49.730194 26233 tablet_copy_source_session.cc:205] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:49.730832 27466 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:49.731295 27466 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap complete.
I20240501 20:11:49.731637 27466 ts_tablet_manager.cc:1367] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: Time spent bootstrapping tablet: real 0.061s	user 0.045s	sys 0.008s
I20240501 20:11:49.732002 27459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f369d7f22f34caaa70a2e167165a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:49.734019 27467 raft_consensus.cc:348] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:49.734009 27466 raft_consensus.cc:348] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:49.734486 27467 raft_consensus.cc:729] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f79d4befcb2c4d6cb0e1b79379164c25, State: Initialized, Role: LEARNER
I20240501 20:11:49.734510 27466 raft_consensus.cc:729] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f79d4befcb2c4d6cb0e1b79379164c25, State: Initialized, Role: LEARNER
I20240501 20:11:49.734956 27467 consensus_queue.cc:260] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:49.734954 27466 consensus_queue.cc:260] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:49.737154 27459 tablet_copy_client.cc:783] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:49.737406 27459 tablet_copy_client.cc:647] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:49.739041 27459 tablet_copy_client.cc:523] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:49.739527 27466 ts_tablet_manager.cc:1398] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I20240501 20:11:49.740733 26364 tablet_copy_service.cc:342] P 1c992b98bb0d4f1b92d058c882fb68b9: Request end of tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-6c4ef284f1844f80aed689e14187638d received from {username='slave'} at 127.8.68.197:52555
I20240501 20:11:49.740906 26364 tablet_copy_service.cc:433] P 1c992b98bb0d4f1b92d058c882fb68b9: ending tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-6c4ef284f1844f80aed689e14187638d on tablet 6c4ef284f1844f80aed689e14187638d with peer f79d4befcb2c4d6cb0e1b79379164c25
I20240501 20:11:49.741914 27467 ts_tablet_manager.cc:1398] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.007s
I20240501 20:11:49.742543 26364 tablet_copy_service.cc:342] P 1c992b98bb0d4f1b92d058c882fb68b9: Request end of tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-57621c56ad05419aaafd7036ced2b691 received from {username='slave'} at 127.8.68.197:52555
I20240501 20:11:49.742671 26364 tablet_copy_service.cc:433] P 1c992b98bb0d4f1b92d058c882fb68b9: ending tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-57621c56ad05419aaafd7036ced2b691 on tablet 57621c56ad05419aaafd7036ced2b691 with peer f79d4befcb2c4d6cb0e1b79379164c25
I20240501 20:11:49.742707 27459 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap starting.
I20240501 20:11:49.775738 27459 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:49.776084 27459 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap complete.
I20240501 20:11:49.776357 27459 ts_tablet_manager.cc:1367] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent bootstrapping tablet: real 0.034s	user 0.032s	sys 0.000s
I20240501 20:11:49.777086 27459 raft_consensus.cc:348] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:49.777287 27459 raft_consensus.cc:729] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0afaccbf1fb84dd7b530d6632c4cd948, State: Initialized, Role: LEARNER
I20240501 20:11:49.777449 27459 consensus_queue.cc:260] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:49.777946 27459 ts_tablet_manager.cc:1398] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:49.778678 26233 tablet_copy_service.cc:342] P 41abd2deb0d7412bbb66b1aa9851d888: Request end of tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-3f369d7f22f34caaa70a2e167165a0d3 received from {username='slave'} at 127.8.68.196:55851
I20240501 20:11:49.778831 26233 tablet_copy_service.cc:433] P 41abd2deb0d7412bbb66b1aa9851d888: ending tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-3f369d7f22f34caaa70a2e167165a0d3 on tablet 3f369d7f22f34caaa70a2e167165a0d3 with peer 0afaccbf1fb84dd7b530d6632c4cd948
I20240501 20:11:49.788978 27467 ts_tablet_manager.cc:902] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: Initiating tablet copy from peer e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067)
I20240501 20:11:49.790423 27467 tablet_copy_client.cc:308] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:41067
I20240501 20:11:49.796120 26099 tablet_copy_service.cc:140] P e9a0d830efeb4d43b13221b36d8e77f0: Received BeginTabletCopySession request for tablet f35f4b0292e045949cae7b7f4120b67c from peer f79d4befcb2c4d6cb0e1b79379164c25 ({username='slave'} at 127.8.68.197:41271)
I20240501 20:11:49.796358 26099 tablet_copy_service.cc:161] P e9a0d830efeb4d43b13221b36d8e77f0: Beginning new tablet copy session on tablet f35f4b0292e045949cae7b7f4120b67c from peer f79d4befcb2c4d6cb0e1b79379164c25 at {username='slave'} at 127.8.68.197:41271: session id = f79d4befcb2c4d6cb0e1b79379164c25-f35f4b0292e045949cae7b7f4120b67c
I20240501 20:11:49.799243 26099 tablet_copy_source_session.cc:205] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:49.800930 27467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35f4b0292e045949cae7b7f4120b67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:49.806425 27467 tablet_copy_client.cc:783] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:49.806703 27467 tablet_copy_client.cc:647] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:49.808355 27467 tablet_copy_client.cc:523] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:49.811872 27467 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap starting.
I20240501 20:11:49.843384 27467 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:49.843711 27467 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap complete.
I20240501 20:11:49.843906 27467 ts_tablet_manager.cc:1367] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: Time spent bootstrapping tablet: real 0.032s	user 0.029s	sys 0.003s
I20240501 20:11:49.844452 27467 raft_consensus.cc:348] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:49.844671 27467 raft_consensus.cc:729] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f79d4befcb2c4d6cb0e1b79379164c25, State: Initialized, Role: LEARNER
I20240501 20:11:49.844841 27467 consensus_queue.cc:260] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:49.845284 27467 ts_tablet_manager.cc:1398] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:49.846046 26099 tablet_copy_service.cc:342] P e9a0d830efeb4d43b13221b36d8e77f0: Request end of tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-f35f4b0292e045949cae7b7f4120b67c received from {username='slave'} at 127.8.68.197:41271
I20240501 20:11:49.846216 26099 tablet_copy_service.cc:433] P e9a0d830efeb4d43b13221b36d8e77f0: ending tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-f35f4b0292e045949cae7b7f4120b67c on tablet f35f4b0292e045949cae7b7f4120b67c with peer f79d4befcb2c4d6cb0e1b79379164c25
I20240501 20:11:50.044031 27019 raft_consensus.cc:1203] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:50.063691 27151 raft_consensus.cc:1203] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:50.071796 27151 raft_consensus.cc:1203] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:50.150991 27019 raft_consensus.cc:1203] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:50.202176 27151 raft_consensus.cc:1203] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:50.221168 27019 raft_consensus.cc:1203] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 20:11:50.468308 27554 raft_consensus.cc:1050] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888: attempting to promote NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 to VOTER
I20240501 20:11:50.469245 27554 consensus_queue.cc:237] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:50.471771 26078 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.471767 26344 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.471871 27151 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.472544 27595 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.472889 27345 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.473169 27554 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.476616 27345 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.477315 26344 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.478000 27151 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.480527 27540 raft_consensus.cc:1050] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9: attempting to promote NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 to VOTER
I20240501 20:11:50.481446 27540 consensus_queue.cc:237] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:50.481412 26078 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.482017 25924 catalog_manager.cc:5484] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:50.484213 27019 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.484318 26212 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.485002 26800 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.487351 26078 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.489136 27600 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.489508 27345 raft_consensus.cc:1050] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888: attempting to promote NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 to VOTER
I20240501 20:11:50.490392 27345 consensus_queue.cc:237] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:50.491966 27600 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.494889 26213 consensus_queue.cc:237] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:50.496644 27151 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.497447 27554 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.498852 27150 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.499337 26344 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.499401 26078 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.499953 26076 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.499948 27345 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.500689 27341 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.501587 27341 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.506655 27540 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.506719 27150 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.510151 27019 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.510741 25926 catalog_manager.cc:5484] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:50.511775 25930 catalog_manager.cc:5484] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 reported cstate change: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.514931 26077 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.518847 26325 tablet_service.cc:1514] Processing DeleteTablet for tablet cd17acc8002948999a5c607ccb9bb189 with delete_type TABLET_DATA_TOMBSTONED (TS 1c992b98bb0d4f1b92d058c882fb68b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41710
I20240501 20:11:50.519443 25920 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd17acc8002948999a5c607ccb9bb189 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:50.519565 27554 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.520077 27554 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.521169 26212 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.523206 26344 consensus_queue.cc:237] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:50.523067 26077 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.531999 27341 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.535648 27019 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.536219 27600 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.536258 26345 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.537066 25926 catalog_manager.cc:5484] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:50.538658 27600 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.540388 26077 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.541565 27600 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.541604 25919 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 57621c56ad05419aaafd7036ced2b691 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:50.542011 27150 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.547432 27018 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.548311 26076 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.547713 27611 tablet_replica.cc:330] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: stopping tablet replica
I20240501 20:11:50.552677 27611 raft_consensus.cc:2229] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:50.553149 27611 raft_consensus.cc:2258] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:50.554673 25926 catalog_manager.cc:5484] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:50.555241 27611 ts_tablet_manager.cc:1875] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:50.556429 26212 consensus_queue.cc:237] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:50.558410 26077 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.558723 27150 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.558568 26191 tablet_service.cc:1514] Processing DeleteTablet for tablet 57621c56ad05419aaafd7036ced2b691 with delete_type TABLET_DATA_TOMBSTONED (TS 41abd2deb0d7412bbb66b1aa9851d888 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59970
I20240501 20:11:50.559083 27345 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.559425 27341 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.561964 27595 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.562621 26077 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.562927 27151 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.567149 25920 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97f145d7f939459fbc804b732189db3f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:50.568854 27618 tablet_replica.cc:330] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888: stopping tablet replica
I20240501 20:11:50.569078 25924 catalog_manager.cc:5484] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:50.569361 27618 raft_consensus.cc:2229] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:50.569499 27611 ts_tablet_manager.cc:1888] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:50.569620 27611 log.cc:1196] T cd17acc8002948999a5c607ccb9bb189 P 1c992b98bb0d4f1b92d058c882fb68b9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/cd17acc8002948999a5c607ccb9bb189
I20240501 20:11:50.569710 27618 raft_consensus.cc:2258] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:50.571169 26076 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.572558 27618 ts_tablet_manager.cc:1875] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:50.575196 25919 catalog_manager.cc:4830] TS 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745): tablet cd17acc8002948999a5c607ccb9bb189 (table rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]) successfully deleted
I20240501 20:11:50.578882 27618 ts_tablet_manager.cc:1888] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:50.579071 27618 log.cc:1196] T 57621c56ad05419aaafd7036ced2b691 P 41abd2deb0d7412bbb66b1aa9851d888: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/57621c56ad05419aaafd7036ced2b691
I20240501 20:11:50.579259 26325 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f145d7f939459fbc804b732189db3f with delete_type TABLET_DATA_TOMBSTONED (TS 1c992b98bb0d4f1b92d058c882fb68b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41710
I20240501 20:11:50.579612 27611 tablet_replica.cc:330] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9: stopping tablet replica
I20240501 20:11:50.579828 27611 raft_consensus.cc:2229] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:50.580098 25920 catalog_manager.cc:4830] TS 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909): tablet 57621c56ad05419aaafd7036ced2b691 (table rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]) successfully deleted
I20240501 20:11:50.580073 27611 raft_consensus.cc:2258] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:50.581022 27611 ts_tablet_manager.cc:1875] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:50.587731 27611 ts_tablet_manager.cc:1888] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:50.587905 27611 log.cc:1196] T 97f145d7f939459fbc804b732189db3f P 1c992b98bb0d4f1b92d058c882fb68b9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/97f145d7f939459fbc804b732189db3f
I20240501 20:11:50.589288 25919 catalog_manager.cc:4830] TS 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745): tablet 97f145d7f939459fbc804b732189db3f (table rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]) successfully deleted
I20240501 20:11:50.614619 27341 raft_consensus.cc:1050] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888: attempting to promote NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 to VOTER
I20240501 20:11:50.615257 27341 consensus_queue.cc:237] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:50.617584 26076 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.617697 26345 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.618113 27150 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.618111 27595 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.618424 27554 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.618682 27345 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.622424 27341 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.623256 26076 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.626595 27150 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.627647 26345 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.630227 25930 catalog_manager.cc:5484] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 4 to 5, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:50.634347 26212 consensus_queue.cc:237] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:50.636476 27150 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.636803 26076 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.637029 27341 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.637902 27341 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.640139 27595 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.642020 26076 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.643596 25920 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f369d7f22f34caaa70a2e167165a0d3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:50.644964 27150 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.646487 25932 catalog_manager.cc:5484] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 reported cstate change: config changed from index 5 to 6, VOTER 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195) evicted. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:50.650079 26325 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f369d7f22f34caaa70a2e167165a0d3 with delete_type TABLET_DATA_TOMBSTONED (TS 1c992b98bb0d4f1b92d058c882fb68b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41710
I20240501 20:11:50.650413 27611 tablet_replica.cc:330] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9: stopping tablet replica
I20240501 20:11:50.650599 27611 raft_consensus.cc:2229] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:50.650956 27611 raft_consensus.cc:2258] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:50.651746 27611 ts_tablet_manager.cc:1875] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:50.657094 27611 ts_tablet_manager.cc:1888] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:50.657222 27611 log.cc:1196] T 3f369d7f22f34caaa70a2e167165a0d3 P 1c992b98bb0d4f1b92d058c882fb68b9: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f369d7f22f34caaa70a2e167165a0d3
I20240501 20:11:50.657866 25919 catalog_manager.cc:4830] TS 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745): tablet 3f369d7f22f34caaa70a2e167165a0d3 (table rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]) successfully deleted
I20240501 20:11:50.719404 27562 raft_consensus.cc:1050] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: attempting to promote NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 to VOTER
I20240501 20:11:50.720191 27562 consensus_queue.cc:237] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:50.722426 27018 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.722719 26212 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.722941 26345 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.722957 27562 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.723304 27674 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.723615 27561 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.727336 27562 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.727962 27018 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.728444 26345 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.732324 25924 catalog_manager.cc:5484] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 reported cstate change: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.732474 26212 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.737237 26076 consensus_queue.cc:237] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:50.738940 27018 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.739091 26345 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.739534 27562 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.740190 27562 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.741572 27561 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.742116 26345 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.743242 27018 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.745133 25920 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f35f4b0292e045949cae7b7f4120b67c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:50.746744 25926 catalog_manager.cc:5484] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.749552 26191 tablet_service.cc:1514] Processing DeleteTablet for tablet f35f4b0292e045949cae7b7f4120b67c with delete_type TABLET_DATA_TOMBSTONED (TS 41abd2deb0d7412bbb66b1aa9851d888 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59970
I20240501 20:11:50.749989 27618 tablet_replica.cc:330] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888: stopping tablet replica
I20240501 20:11:50.750170 27618 raft_consensus.cc:2229] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:50.750509 27618 raft_consensus.cc:2258] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:50.751346 27618 ts_tablet_manager.cc:1875] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:50.755477 27618 ts_tablet_manager.cc:1888] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:50.755571 27618 log.cc:1196] T f35f4b0292e045949cae7b7f4120b67c P 41abd2deb0d7412bbb66b1aa9851d888: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/f35f4b0292e045949cae7b7f4120b67c
I20240501 20:11:50.756119 25920 catalog_manager.cc:4830] TS 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909): tablet f35f4b0292e045949cae7b7f4120b67c (table rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]) successfully deleted
I20240501 20:11:50.774798 27600 raft_consensus.cc:1050] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9: attempting to promote NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 to VOTER
I20240501 20:11:50.775393 27600 consensus_queue.cc:237] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:50.777338 27018 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.777382 26076 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.777412 26212 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:50.777856 27600 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.778124 26800 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.778373 27540 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:50.780845 27600 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.781462 26076 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.782213 27018 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.785775 26212 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.786027 25932 catalog_manager.cc:5484] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 4 to 5, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:50.789811 26345 consensus_queue.cc:237] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:50.791734 27018 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.791944 26076 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:50.792263 27600 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.793107 27600 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:50.794236 27540 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.795691 26076 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.796811 25919 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c4ef284f1844f80aed689e14187638d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:50.797554 27018 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:50.798250 25926 catalog_manager.cc:5484] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 5 to 6, VOTER 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194) evicted. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:50.801615 26191 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4ef284f1844f80aed689e14187638d with delete_type TABLET_DATA_TOMBSTONED (TS 41abd2deb0d7412bbb66b1aa9851d888 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59970
I20240501 20:11:50.801960 27618 tablet_replica.cc:330] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888: stopping tablet replica
I20240501 20:11:50.802139 27618 raft_consensus.cc:2229] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:50.802417 27618 raft_consensus.cc:2258] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:50.803181 27618 ts_tablet_manager.cc:1875] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:50.807193 27618 ts_tablet_manager.cc:1888] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:50.807286 27618 log.cc:1196] T 6c4ef284f1844f80aed689e14187638d P 41abd2deb0d7412bbb66b1aa9851d888: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-1/wal/wals/6c4ef284f1844f80aed689e14187638d
I20240501 20:11:50.807806 25920 catalog_manager.cc:4830] TS 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909): tablet 6c4ef284f1844f80aed689e14187638d (table rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]) successfully deleted
I20240501 20:11:50.995244 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:51.004560 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:51.020251 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:51.035356 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:51.036504 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:51.205726 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:51.210568 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:51.218938 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:51.228128 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:51.228940 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:51.362586 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:51.369056 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:51.378914 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:51.383184 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:51.392880 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:51.926200 25932 ts_manager.cc:292] Set tserver state for e9a0d830efeb4d43b13221b36d8e77f0 to MAINTENANCE_MODE
I20240501 20:11:52.456557 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.459221 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.461673 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:52.465154 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:52.485204 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:52.633436 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.637830 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:52.643759 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:52.646503 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:52.654552 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.813205 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.824283 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:52.824697 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:52.829741 26056 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:52.848322 26999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:52.902004 26345 consensus_queue.cc:237] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:52.903903 27019 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:52.904096 26076 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:52.904424 27805 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:52.904767 27916 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:52.907723 27805 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:52.908334 27019 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:52.908337 26076 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:52.912720 25932 catalog_manager.cc:5484] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:52.924011 26076 consensus_queue.cc:237] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:52.926085 27019 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:52.926432 26345 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:52.926735 27562 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:52.927312 27562 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:52.928323 26345 consensus_queue.cc:237] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:52.930945 27018 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:52.931752 27916 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:52.932899 27562 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:52.933488 26076 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:52.934247 27805 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:52.934055 26344 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:52.934566 27018 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:52.934991 27805 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:52.938437 25919 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 57621c56ad05419aaafd7036ced2b691 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:11:52.938985 25926 catalog_manager.cc:5484] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:52.940961 27018 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:52.941570 26076 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:52.942107 25924 catalog_manager.cc:5484] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:52.944566 26077 consensus_queue.cc:237] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:52.948943 26344 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:52.949407 27018 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:52.949887 27930 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:52.950477 27930 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:52.954097 27931 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:52.954550 26344 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
W20240501 20:11:52.956372 26279 consensus_peers.cc:485] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 -> Peer 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631): Couldn't send request to peer 0afaccbf1fb84dd7b530d6632c4cd948. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57621c56ad05419aaafd7036ced2b691. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:52.958983 27018 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:52.959311 25932 catalog_manager.cc:5484] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:52.961853 25920 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f35f4b0292e045949cae7b7f4120b67c with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240501 20:11:52.972177 26011 consensus_peers.cc:485] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 -> Peer 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631): Couldn't send request to peer 0afaccbf1fb84dd7b530d6632c4cd948. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f35f4b0292e045949cae7b7f4120b67c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:52.973212 26212 consensus_queue.cc:237] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:52.975200 26077 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:52.975229 27150 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:52.975647 27554 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:52.975988 27943 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:52.978704 27944 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:52.979251 26077 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:52.979512 27150 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:52.985518 25932 catalog_manager.cc:5484] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:52.992789 26212 consensus_queue.cc:237] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:52.999366 26077 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:53.000126 27150 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:53.000685 27554 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:53.001011 27943 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:53.003785 27554 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
W20240501 20:11:53.006379 26147 consensus_peers.cc:485] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 -> Peer f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545): Couldn't send request to peer f79d4befcb2c4d6cb0e1b79379164c25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f369d7f22f34caaa70a2e167165a0d3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:53.006908 26077 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:53.007090 27150 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:53.007544 25920 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3f369d7f22f34caaa70a2e167165a0d3 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:11:53.009735 25926 catalog_manager.cc:5484] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:53.530643 27995 ts_tablet_manager.cc:902] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: Initiating tablet copy from peer e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067)
I20240501 20:11:53.531769 27995 tablet_copy_client.cc:308] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.193:41067
I20240501 20:11:53.537184 26099 tablet_copy_service.cc:140] P e9a0d830efeb4d43b13221b36d8e77f0: Received BeginTabletCopySession request for tablet f35f4b0292e045949cae7b7f4120b67c from peer 0afaccbf1fb84dd7b530d6632c4cd948 ({username='slave'} at 127.8.68.196:60435)
I20240501 20:11:53.537331 26099 tablet_copy_service.cc:161] P e9a0d830efeb4d43b13221b36d8e77f0: Beginning new tablet copy session on tablet f35f4b0292e045949cae7b7f4120b67c from peer 0afaccbf1fb84dd7b530d6632c4cd948 at {username='slave'} at 127.8.68.196:60435: session id = 0afaccbf1fb84dd7b530d6632c4cd948-f35f4b0292e045949cae7b7f4120b67c
I20240501 20:11:53.538920 28001 ts_tablet_manager.cc:902] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: Initiating tablet copy from peer 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:53.539409 26099 tablet_copy_source_session.cc:205] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:53.540484 28001 tablet_copy_client.cc:308] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:37745
I20240501 20:11:53.540758 27995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35f4b0292e045949cae7b7f4120b67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.546317 26364 tablet_copy_service.cc:140] P 1c992b98bb0d4f1b92d058c882fb68b9: Received BeginTabletCopySession request for tablet 57621c56ad05419aaafd7036ced2b691 from peer 0afaccbf1fb84dd7b530d6632c4cd948 ({username='slave'} at 127.8.68.196:36913)
I20240501 20:11:53.546502 26364 tablet_copy_service.cc:161] P 1c992b98bb0d4f1b92d058c882fb68b9: Beginning new tablet copy session on tablet 57621c56ad05419aaafd7036ced2b691 from peer 0afaccbf1fb84dd7b530d6632c4cd948 at {username='slave'} at 127.8.68.196:36913: session id = 0afaccbf1fb84dd7b530d6632c4cd948-57621c56ad05419aaafd7036ced2b691
I20240501 20:11:53.546543 27995 tablet_copy_client.cc:783] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:53.546749 27995 tablet_copy_client.cc:647] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:53.548238 27995 tablet_copy_client.cc:523] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:53.548652 26364 tablet_copy_source_session.cc:205] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:53.549803 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57621c56ad05419aaafd7036ced2b691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.550843 27995 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap starting.
I20240501 20:11:53.553913 28001 tablet_copy_client.cc:783] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:53.554126 28001 tablet_copy_client.cc:647] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:53.555680 28001 tablet_copy_client.cc:523] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:53.557907 28001 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap starting.
I20240501 20:11:53.569919 28008 ts_tablet_manager.cc:902] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: Initiating tablet copy from peer 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:53.592973 27995 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:53.593828 27995 tablet_bootstrap.cc:492] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap complete.
I20240501 20:11:53.594180 27995 ts_tablet_manager.cc:1367] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent bootstrapping tablet: real 0.043s	user 0.037s	sys 0.006s
I20240501 20:11:53.595043 27995 raft_consensus.cc:348] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:53.595391 27995 raft_consensus.cc:729] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0afaccbf1fb84dd7b530d6632c4cd948, State: Initialized, Role: LEARNER
I20240501 20:11:53.595621 27995 consensus_queue.cc:260] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:53.608340 27995 ts_tablet_manager.cc:1398] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.004s
I20240501 20:11:53.609437 26099 tablet_copy_service.cc:342] P e9a0d830efeb4d43b13221b36d8e77f0: Request end of tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-f35f4b0292e045949cae7b7f4120b67c received from {username='slave'} at 127.8.68.196:60435
I20240501 20:11:53.609572 26099 tablet_copy_service.cc:433] P e9a0d830efeb4d43b13221b36d8e77f0: ending tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-f35f4b0292e045949cae7b7f4120b67c on tablet f35f4b0292e045949cae7b7f4120b67c with peer 0afaccbf1fb84dd7b530d6632c4cd948
I20240501 20:11:53.614653 28008 tablet_copy_client.cc:308] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:35909
I20240501 20:11:53.617813 28001 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:53.618199 28001 tablet_bootstrap.cc:492] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap complete.
I20240501 20:11:53.618474 28001 ts_tablet_manager.cc:1367] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent bootstrapping tablet: real 0.061s	user 0.025s	sys 0.019s
I20240501 20:11:53.619164 28001 raft_consensus.cc:348] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:53.619421 28001 raft_consensus.cc:729] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0afaccbf1fb84dd7b530d6632c4cd948, State: Initialized, Role: LEARNER
I20240501 20:11:53.619602 28001 consensus_queue.cc:260] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:53.620136 28001 ts_tablet_manager.cc:1398] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 20:11:53.620869 26364 tablet_copy_service.cc:342] P 1c992b98bb0d4f1b92d058c882fb68b9: Request end of tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-57621c56ad05419aaafd7036ced2b691 received from {username='slave'} at 127.8.68.196:36913
I20240501 20:11:53.620995 26364 tablet_copy_service.cc:433] P 1c992b98bb0d4f1b92d058c882fb68b9: ending tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-57621c56ad05419aaafd7036ced2b691 on tablet 57621c56ad05419aaafd7036ced2b691 with peer 0afaccbf1fb84dd7b530d6632c4cd948
I20240501 20:11:53.621656 26233 tablet_copy_service.cc:140] P 41abd2deb0d7412bbb66b1aa9851d888: Received BeginTabletCopySession request for tablet 3f369d7f22f34caaa70a2e167165a0d3 from peer f79d4befcb2c4d6cb0e1b79379164c25 ({username='slave'} at 127.8.68.197:38227)
I20240501 20:11:53.621824 26233 tablet_copy_service.cc:161] P 41abd2deb0d7412bbb66b1aa9851d888: Beginning new tablet copy session on tablet 3f369d7f22f34caaa70a2e167165a0d3 from peer f79d4befcb2c4d6cb0e1b79379164c25 at {username='slave'} at 127.8.68.197:38227: session id = f79d4befcb2c4d6cb0e1b79379164c25-3f369d7f22f34caaa70a2e167165a0d3
I20240501 20:11:53.624501 26233 tablet_copy_source_session.cc:205] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:53.625597 28008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f369d7f22f34caaa70a2e167165a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.629709 28008 tablet_copy_client.cc:783] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:53.629926 28008 tablet_copy_client.cc:647] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:53.631420 28008 tablet_copy_client.cc:523] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:53.633826 28008 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap starting.
I20240501 20:11:53.665894 28008 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:53.666208 28008 tablet_bootstrap.cc:492] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap complete.
I20240501 20:11:53.666436 28008 ts_tablet_manager.cc:1367] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: Time spent bootstrapping tablet: real 0.033s	user 0.030s	sys 0.005s
I20240501 20:11:53.667034 28008 raft_consensus.cc:348] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:53.667225 28008 raft_consensus.cc:729] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f79d4befcb2c4d6cb0e1b79379164c25, State: Initialized, Role: LEARNER
I20240501 20:11:53.667362 28008 consensus_queue.cc:260] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:53.667999 28008 ts_tablet_manager.cc:1398] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.001s
I20240501 20:11:53.668643 26233 tablet_copy_service.cc:342] P 41abd2deb0d7412bbb66b1aa9851d888: Request end of tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-3f369d7f22f34caaa70a2e167165a0d3 received from {username='slave'} at 127.8.68.197:38227
I20240501 20:11:53.668756 26233 tablet_copy_service.cc:433] P 41abd2deb0d7412bbb66b1aa9851d888: ending tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-3f369d7f22f34caaa70a2e167165a0d3 on tablet 3f369d7f22f34caaa70a2e167165a0d3 with peer f79d4befcb2c4d6cb0e1b79379164c25
I20240501 20:11:53.913534 27150 raft_consensus.cc:1203] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:53.991042 27150 raft_consensus.cc:1203] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:54.030802 27018 raft_consensus.cc:1203] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:54.353899 28033 raft_consensus.cc:1050] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: attempting to promote NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 to VOTER
I20240501 20:11:54.354663 28033 consensus_queue.cc:237] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:54.356869 27150 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:54.357002 27018 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:54.357085 26344 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer e9a0d830efeb4d43b13221b36d8e77f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:54.357496 28069 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:54.357890 28033 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:54.358204 27930 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:54.371457 27930 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.372190 27018 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.372674 26344 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.376616 25924 catalog_manager.cc:5484] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 reported cstate change: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.376746 27150 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.382392 28049 raft_consensus.cc:1050] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9: attempting to promote NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 to VOTER
I20240501 20:11:54.383164 28049 consensus_queue.cc:237] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:54.386746 27018 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:54.386893 26076 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:54.387394 28050 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:54.387913 27805 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:54.388240 27150 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:54.389465 28050 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:54.392060 28049 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.392242 27018 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.396557 25932 catalog_manager.cc:5484] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 reported cstate change: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.396557 26076 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.399894 27150 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.406085 26344 consensus_queue.cc:237] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:54.407892 27150 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:54.408099 27018 raft_consensus.cc:1261] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:54.408475 28088 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:54.409236 28088 consensus_queue.cc:1035] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:54.411533 28049 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.412086 27150 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.412384 27018 raft_consensus.cc:2932] T 57621c56ad05419aaafd7036ced2b691 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.415580 25919 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 57621c56ad05419aaafd7036ced2b691 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:54.416517 25932 catalog_manager.cc:5484] T 57621c56ad05419aaafd7036ced2b691 P 0afaccbf1fb84dd7b530d6632c4cd948 reported cstate change: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:54.421059 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet 57621c56ad05419aaafd7036ced2b691 with delete_type TABLET_DATA_TOMBSTONED (TS e9a0d830efeb4d43b13221b36d8e77f0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57318
I20240501 20:11:54.422127 28099 tablet_replica.cc:330] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0: stopping tablet replica
I20240501 20:11:54.422555 28099 raft_consensus.cc:2229] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:54.422982 28099 raft_consensus.cc:2258] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:54.425406 28099 ts_tablet_manager.cc:1875] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:54.430469 28099 ts_tablet_manager.cc:1888] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:11:54.430696 28099 log.cc:1196] T 57621c56ad05419aaafd7036ced2b691 P e9a0d830efeb4d43b13221b36d8e77f0: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/57621c56ad05419aaafd7036ced2b691
I20240501 20:11:54.431535 25920 catalog_manager.cc:4830] TS e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067): tablet 57621c56ad05419aaafd7036ced2b691 (table rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]) successfully deleted
I20240501 20:11:54.448724 26076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c"
dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35788
I20240501 20:11:54.448936 26076 raft_consensus.cc:595] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:54.638270 28105 raft_consensus.cc:1050] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888: attempting to promote NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 to VOTER
I20240501 20:11:54.639011 28105 consensus_queue.cc:237] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:54.641093 27018 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:54.641158 27150 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:54.641213 26076 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:54.641705 28052 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:54.642028 28105 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:54.642361 28107 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:54.645198 28052 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.645857 26076 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.649334 27150 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.650213 25932 catalog_manager.cc:5484] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 reported cstate change: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.650799 27018 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.658319 26212 consensus_queue.cc:237] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:54.660073 27018 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:54.660120 27150 raft_consensus.cc:1261] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:54.661372 28052 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:54.661886 28052 consensus_queue.cc:1035] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:54.663776 28052 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.664696 27150 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.666872 27018 raft_consensus.cc:2932] T 3f369d7f22f34caaa70a2e167165a0d3 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.667392 25920 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f369d7f22f34caaa70a2e167165a0d3 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:54.669054 25930 catalog_manager.cc:5484] T 3f369d7f22f34caaa70a2e167165a0d3 P 0afaccbf1fb84dd7b530d6632c4cd948 reported cstate change: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.672225 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f369d7f22f34caaa70a2e167165a0d3 with delete_type TABLET_DATA_TOMBSTONED (TS e9a0d830efeb4d43b13221b36d8e77f0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57318
I20240501 20:11:54.672560 28099 tablet_replica.cc:330] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0: stopping tablet replica
I20240501 20:11:54.672752 28099 raft_consensus.cc:2229] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:54.673190 28099 raft_consensus.cc:2258] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:54.673970 28099 ts_tablet_manager.cc:1875] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:54.680240 28099 ts_tablet_manager.cc:1888] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:11:54.680343 28099 log.cc:1196] T 3f369d7f22f34caaa70a2e167165a0d3 P e9a0d830efeb4d43b13221b36d8e77f0: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/3f369d7f22f34caaa70a2e167165a0d3
I20240501 20:11:54.680979 25920 catalog_manager.cc:4830] TS e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067): tablet 3f369d7f22f34caaa70a2e167165a0d3 (table rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]) successfully deleted
I20240501 20:11:54.701437 26076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c"
dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35806
I20240501 20:11:54.701607 26076 raft_consensus.cc:595] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:54.792850 28069 raft_consensus.cc:979] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: : Instructing follower f79d4befcb2c4d6cb0e1b79379164c25 to start an election
I20240501 20:11:54.793033 28069 raft_consensus.cc:1067] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Signalling peer f79d4befcb2c4d6cb0e1b79379164c25 to start an election
I20240501 20:11:54.793903 27018 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c"
dest_uuid: "f79d4befcb2c4d6cb0e1b79379164c25"
 from {username='slave'} at 127.8.68.193:58615
I20240501 20:11:54.794221 27018 raft_consensus.cc:482] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:54.794418 27018 raft_consensus.cc:3037] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:54.796583 27018 raft_consensus.cc:504] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:54.798035 27018 leader_election.cc:288] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [CANDIDATE]: Term 2 election: Requested vote from peers 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745), e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067), 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631)
I20240501 20:11:54.798409 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c" candidate_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "1c992b98bb0d4f1b92d058c882fb68b9"
I20240501 20:11:54.798619 26344 raft_consensus.cc:3037] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:54.798602 26076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c" candidate_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e9a0d830efeb4d43b13221b36d8e77f0"
I20240501 20:11:54.798843 26076 raft_consensus.cc:3032] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:54.798923 26076 raft_consensus.cc:729] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9a0d830efeb4d43b13221b36d8e77f0, State: Running, Role: LEADER
I20240501 20:11:54.799240 26076 consensus_queue.cc:260] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:54.799717 26076 raft_consensus.cc:3037] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:54.801705 26344 raft_consensus.cc:2454] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f79d4befcb2c4d6cb0e1b79379164c25 in term 2.
I20240501 20:11:54.802485 26076 raft_consensus.cc:2454] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f79d4befcb2c4d6cb0e1b79379164c25 in term 2.
I20240501 20:11:54.803138 26955 leader_election.cc:302] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c992b98bb0d4f1b92d058c882fb68b9, e9a0d830efeb4d43b13221b36d8e77f0, f79d4befcb2c4d6cb0e1b79379164c25; no voters: 
I20240501 20:11:54.803531 28016 raft_consensus.cc:2781] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:54.804025 28016 raft_consensus.cc:686] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 2 LEADER]: Becoming Leader. State: Replica: f79d4befcb2c4d6cb0e1b79379164c25, State: Running, Role: LEADER
I20240501 20:11:54.804549 28016 consensus_queue.cc:237] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:54.808341 25924 catalog_manager.cc:5484] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 reported cstate change: term changed from 1 to 2, leader changed from e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) to f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197). New cstate: current_term: 2 leader_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:54.809161 27150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35f4b0292e045949cae7b7f4120b67c" candidate_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "0afaccbf1fb84dd7b530d6632c4cd948"
I20240501 20:11:54.809507 27150 raft_consensus.cc:3037] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:54.811914 27150 raft_consensus.cc:2454] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f79d4befcb2c4d6cb0e1b79379164c25 in term 2.
I20240501 20:11:54.912390 26344 consensus_queue.cc:237] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:54.914425 26212 raft_consensus.cc:1261] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:54.914453 26076 raft_consensus.cc:1261] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:54.914932 28088 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.915200 28049 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:54.917728 28088 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:54.918161 26212 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:54.918478 26076 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:54.922314 25924 catalog_manager.cc:5484] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:54.927582 26344 consensus_queue.cc:237] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
W20240501 20:11:54.929905 26280 consensus_peers.cc:485] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 -> Peer f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545): Couldn't send request to peer f79d4befcb2c4d6cb0e1b79379164c25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5c251d108ea4e7f85f56173c24fdec4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:54.930024 26076 raft_consensus.cc:1261] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:54.930377 26212 raft_consensus.cc:1261] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:54.931645 28146 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:54.931950 28049 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:54.934350 28049 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:54.934955 26212 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:54.935839 26344 consensus_queue.cc:237] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:54.934955 26076 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:54.937695 25919 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a5c251d108ea4e7f85f56173c24fdec4 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:11:54.938916 25926 catalog_manager.cc:5484] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 4 to 5, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:54.944278 26076 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:54.944272 27018 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:54.944882 28088 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:54.945595 28049 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:54.947428 28088 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.948017 27018 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.948408 26076 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.952411 25924 catalog_manager.cc:5484] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:54.957255 26344 consensus_queue.cc:237] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
W20240501 20:11:54.959555 26279 consensus_peers.cc:485] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 -> Peer 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631): Couldn't send request to peer 0afaccbf1fb84dd7b530d6632c4cd948. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c4ef284f1844f80aed689e14187638d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:54.959810 27018 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:54.960391 28146 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:54.961434 26076 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:54.962414 28146 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:54.963379 28088 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:54.964417 27018 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:54.965704 26076 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:54.966917 25919 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6c4ef284f1844f80aed689e14187638d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:11:54.969142 25924 catalog_manager.cc:5484] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 7 to 8, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:55.227241 26077 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 2 FOLLOWER]: Refusing update from remote peer f79d4befcb2c4d6cb0e1b79379164c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:55.227804 28016 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:55.256923 27150 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 2 FOLLOWER]: Refusing update from remote peer f79d4befcb2c4d6cb0e1b79379164c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:55.257406 28016 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:55.262496 26344 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 2 FOLLOWER]: Refusing update from remote peer f79d4befcb2c4d6cb0e1b79379164c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:55.263126 28191 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240501 20:11:55.355583 28195 ts_tablet_manager.cc:902] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: Initiating tablet copy from peer 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:55.356909 28195 tablet_copy_client.cc:308] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:37745
I20240501 20:11:55.357547 26364 tablet_copy_service.cc:140] P 1c992b98bb0d4f1b92d058c882fb68b9: Received BeginTabletCopySession request for tablet 6c4ef284f1844f80aed689e14187638d from peer 0afaccbf1fb84dd7b530d6632c4cd948 ({username='slave'} at 127.8.68.196:36913)
I20240501 20:11:55.357725 26364 tablet_copy_service.cc:161] P 1c992b98bb0d4f1b92d058c882fb68b9: Beginning new tablet copy session on tablet 6c4ef284f1844f80aed689e14187638d from peer 0afaccbf1fb84dd7b530d6632c4cd948 at {username='slave'} at 127.8.68.196:36913: session id = 0afaccbf1fb84dd7b530d6632c4cd948-6c4ef284f1844f80aed689e14187638d
I20240501 20:11:55.359787 26364 tablet_copy_source_session.cc:205] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:55.360973 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4ef284f1844f80aed689e14187638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:55.365197 28195 tablet_copy_client.cc:783] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:55.365412 28195 tablet_copy_client.cc:647] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:55.366905 28195 tablet_copy_client.cc:523] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:55.369068 28195 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap starting.
I20240501 20:11:55.399782 28195 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:55.400125 28195 tablet_bootstrap.cc:492] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap complete.
I20240501 20:11:55.400360 28195 ts_tablet_manager.cc:1367] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent bootstrapping tablet: real 0.031s	user 0.026s	sys 0.005s
I20240501 20:11:55.400946 28195 raft_consensus.cc:348] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:55.401150 28195 raft_consensus.cc:729] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0afaccbf1fb84dd7b530d6632c4cd948, State: Initialized, Role: LEARNER
I20240501 20:11:55.401337 28195 consensus_queue.cc:260] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:55.401762 28195 ts_tablet_manager.cc:1398] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:55.402422 26364 tablet_copy_service.cc:342] P 1c992b98bb0d4f1b92d058c882fb68b9: Request end of tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-6c4ef284f1844f80aed689e14187638d received from {username='slave'} at 127.8.68.196:36913
I20240501 20:11:55.402581 26364 tablet_copy_service.cc:433] P 1c992b98bb0d4f1b92d058c882fb68b9: ending tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-6c4ef284f1844f80aed689e14187638d on tablet 6c4ef284f1844f80aed689e14187638d with peer 0afaccbf1fb84dd7b530d6632c4cd948
I20240501 20:11:55.417364 28202 ts_tablet_manager.cc:902] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: Initiating tablet copy from peer 1c992b98bb0d4f1b92d058c882fb68b9 (127.8.68.195:37745)
I20240501 20:11:55.418639 28202 tablet_copy_client.cc:308] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.195:37745
I20240501 20:11:55.419281 26364 tablet_copy_service.cc:140] P 1c992b98bb0d4f1b92d058c882fb68b9: Received BeginTabletCopySession request for tablet a5c251d108ea4e7f85f56173c24fdec4 from peer f79d4befcb2c4d6cb0e1b79379164c25 ({username='slave'} at 127.8.68.197:52555)
I20240501 20:11:55.419423 26364 tablet_copy_service.cc:161] P 1c992b98bb0d4f1b92d058c882fb68b9: Beginning new tablet copy session on tablet a5c251d108ea4e7f85f56173c24fdec4 from peer f79d4befcb2c4d6cb0e1b79379164c25 at {username='slave'} at 127.8.68.197:52555: session id = f79d4befcb2c4d6cb0e1b79379164c25-a5c251d108ea4e7f85f56173c24fdec4
I20240501 20:11:55.421262 26364 tablet_copy_source_session.cc:205] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:55.422240 28202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c251d108ea4e7f85f56173c24fdec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:55.427340 28202 tablet_copy_client.cc:783] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:55.427641 28202 tablet_copy_client.cc:647] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:55.429512 28202 tablet_copy_client.cc:523] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:55.433444 28202 tablet_bootstrap.cc:492] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap starting.
I20240501 20:11:55.471621 28202 tablet_bootstrap.cc:492] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:55.472054 28202 tablet_bootstrap.cc:492] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap complete.
I20240501 20:11:55.472323 28202 ts_tablet_manager.cc:1367] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: Time spent bootstrapping tablet: real 0.039s	user 0.038s	sys 0.002s
I20240501 20:11:55.472898 28202 raft_consensus.cc:348] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:55.473094 28202 raft_consensus.cc:729] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f79d4befcb2c4d6cb0e1b79379164c25, State: Initialized, Role: LEARNER
I20240501 20:11:55.473243 28202 consensus_queue.cc:260] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:55.473676 28202 ts_tablet_manager.cc:1398] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:55.474399 26364 tablet_copy_service.cc:342] P 1c992b98bb0d4f1b92d058c882fb68b9: Request end of tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-a5c251d108ea4e7f85f56173c24fdec4 received from {username='slave'} at 127.8.68.197:52555
I20240501 20:11:55.474515 26364 tablet_copy_service.cc:433] P 1c992b98bb0d4f1b92d058c882fb68b9: ending tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-a5c251d108ea4e7f85f56173c24fdec4 on tablet a5c251d108ea4e7f85f56173c24fdec4 with peer f79d4befcb2c4d6cb0e1b79379164c25
I20240501 20:11:55.763605 27151 raft_consensus.cc:1203] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:55.861482 27018 consensus_queue.cc:237] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:55.863272 27151 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 2 FOLLOWER]: Refusing update from remote peer f79d4befcb2c4d6cb0e1b79379164c25: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:11:55.863451 26344 raft_consensus.cc:1261] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 2 FOLLOWER]: Refusing update from remote peer f79d4befcb2c4d6cb0e1b79379164c25: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:11:55.863819 28241 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:11:55.864169 28242 consensus_queue.cc:1035] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:11:55.866175 28241 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P f79d4befcb2c4d6cb0e1b79379164c25 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:55.866767 26344 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:55.867033 27151 raft_consensus.cc:2932] T f35f4b0292e045949cae7b7f4120b67c P 0afaccbf1fb84dd7b530d6632c4cd948 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:55.870888 25919 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f35f4b0292e045949cae7b7f4120b67c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:55.871708 25930 catalog_manager.cc:5484] T f35f4b0292e045949cae7b7f4120b67c P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 9 to 11, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New cstate: current_term: 2 leader_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:55.876031 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet f35f4b0292e045949cae7b7f4120b67c with delete_type TABLET_DATA_TOMBSTONED (TS e9a0d830efeb4d43b13221b36d8e77f0 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:57318
I20240501 20:11:55.876650 28247 tablet_replica.cc:330] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: stopping tablet replica
I20240501 20:11:55.876806 28247 raft_consensus.cc:2229] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:55.876964 28247 raft_consensus.cc:2258] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:55.877650 28247 ts_tablet_manager.cc:1875] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:55.881662 28247 ts_tablet_manager.cc:1888] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 20:11:55.881767 28247 log.cc:1196] T f35f4b0292e045949cae7b7f4120b67c P e9a0d830efeb4d43b13221b36d8e77f0: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/f35f4b0292e045949cae7b7f4120b67c
I20240501 20:11:55.882395 25920 catalog_manager.cc:4830] TS e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067): tablet f35f4b0292e045949cae7b7f4120b67c (table rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]) successfully deleted
I20240501 20:11:55.967909 27018 raft_consensus.cc:1203] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 20:11:56.123230 26212 consensus_queue.cc:237] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:56.124976 27151 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:56.125497 26077 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:56.125469 28270 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:56.126032 28270 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:56.128209 28270 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.128836 27151 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.129599 26077 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.133460 25924 catalog_manager.cc:5484] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:56.139254 26212 consensus_queue.cc:237] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:56.141589 27151 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:11:56.141741 26147 consensus_peers.cc:485] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 -> Peer f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545): Couldn't send request to peer f79d4befcb2c4d6cb0e1b79379164c25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97f145d7f939459fbc804b732189db3f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:56.142156 28052 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:56.142601 26076 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:56.143237 28052 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:56.144827 28270 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:56.145282 27151 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:56.146044 26077 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:56.148411 25920 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 97f145d7f939459fbc804b732189db3f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:11:56.149338 25924 catalog_manager.cc:5484] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 reported cstate change: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:56.372344 28049 raft_consensus.cc:1050] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9: attempting to promote NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 to VOTER
I20240501 20:11:56.373016 28049 consensus_queue.cc:237] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:56.375195 26077 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:56.375319 27151 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:56.375309 27018 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:56.375829 28292 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:56.376231 28049 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:56.376469 28240 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:56.380671 28240 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.381304 26077 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.385277 27151 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.385552 25926 catalog_manager.cc:5484] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 reported cstate change: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.393958 27018 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.396593 26344 consensus_queue.cc:237] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:56.398497 27018 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:56.398905 27151 raft_consensus.cc:1261] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:56.399009 28049 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:56.399509 28240 consensus_queue.cc:1035] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:56.401041 28240 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.401611 27019 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.403645 25919 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c4ef284f1844f80aed689e14187638d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:56.404963 27150 raft_consensus.cc:2932] T 6c4ef284f1844f80aed689e14187638d P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.405232 25924 catalog_manager.cc:5484] T 6c4ef284f1844f80aed689e14187638d P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:56.408855 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4ef284f1844f80aed689e14187638d with delete_type TABLET_DATA_TOMBSTONED (TS e9a0d830efeb4d43b13221b36d8e77f0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57318
I20240501 20:11:56.409405 28304 tablet_replica.cc:330] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0: stopping tablet replica
I20240501 20:11:56.409586 28304 raft_consensus.cc:2229] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:56.409749 28304 raft_consensus.cc:2258] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:56.410642 28304 ts_tablet_manager.cc:1875] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:56.416429 28240 raft_consensus.cc:1050] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9: attempting to promote NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 to VOTER
I20240501 20:11:56.416743 28304 ts_tablet_manager.cc:1888] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:11:56.416852 28304 log.cc:1196] T 6c4ef284f1844f80aed689e14187638d P e9a0d830efeb4d43b13221b36d8e77f0: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/6c4ef284f1844f80aed689e14187638d
I20240501 20:11:56.417151 28240 consensus_queue.cc:237] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:56.417528 25920 catalog_manager.cc:4830] TS e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067): tablet 6c4ef284f1844f80aed689e14187638d (table rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]) successfully deleted
I20240501 20:11:56.419252 26212 raft_consensus.cc:1261] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:56.419358 26077 raft_consensus.cc:1261] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:56.419763 28292 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:56.420058 27018 raft_consensus.cc:1261] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:56.420047 28049 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:56.420500 28088 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:56.435221 28292 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:56.438953 27019 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:56.440441 26077 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:56.441215 25924 catalog_manager.cc:5484] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:56.446959 26212 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:56.449187 26344 consensus_queue.cc:237] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:56.451809 26212 raft_consensus.cc:1261] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:56.452673 28292 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:56.453341 27019 raft_consensus.cc:1261] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 1c992b98bb0d4f1b92d058c882fb68b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:56.453912 28292 consensus_queue.cc:1035] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:56.455034 28292 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:56.457718 25919 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a5c251d108ea4e7f85f56173c24fdec4 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:56.458679 26212 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:56.460091 27018 raft_consensus.cc:2932] T a5c251d108ea4e7f85f56173c24fdec4 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:56.464203 25926 catalog_manager.cc:5484] T a5c251d108ea4e7f85f56173c24fdec4 P 1c992b98bb0d4f1b92d058c882fb68b9 reported cstate change: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:56.467751 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c251d108ea4e7f85f56173c24fdec4 with delete_type TABLET_DATA_TOMBSTONED (TS e9a0d830efeb4d43b13221b36d8e77f0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57318
I20240501 20:11:56.468089 28304 tablet_replica.cc:330] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: stopping tablet replica
I20240501 20:11:56.468278 28304 raft_consensus.cc:2229] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:56.468478 28304 raft_consensus.cc:2258] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:56.470587 28304 ts_tablet_manager.cc:1875] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:56.475363 28304 ts_tablet_manager.cc:1888] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:11:56.475467 28304 log.cc:1196] T a5c251d108ea4e7f85f56173c24fdec4 P e9a0d830efeb4d43b13221b36d8e77f0: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/a5c251d108ea4e7f85f56173c24fdec4
I20240501 20:11:56.476037 25920 catalog_manager.cc:4830] TS e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067): tablet a5c251d108ea4e7f85f56173c24fdec4 (table rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]) successfully deleted
I20240501 20:11:56.675838 28316 ts_tablet_manager.cc:902] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: Initiating tablet copy from peer 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:56.677428 28316 tablet_copy_client.cc:308] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:35909
I20240501 20:11:56.678179 26233 tablet_copy_service.cc:140] P 41abd2deb0d7412bbb66b1aa9851d888: Received BeginTabletCopySession request for tablet 97f145d7f939459fbc804b732189db3f from peer f79d4befcb2c4d6cb0e1b79379164c25 ({username='slave'} at 127.8.68.197:38227)
I20240501 20:11:56.678372 26233 tablet_copy_service.cc:161] P 41abd2deb0d7412bbb66b1aa9851d888: Beginning new tablet copy session on tablet 97f145d7f939459fbc804b732189db3f from peer f79d4befcb2c4d6cb0e1b79379164c25 at {username='slave'} at 127.8.68.197:38227: session id = f79d4befcb2c4d6cb0e1b79379164c25-97f145d7f939459fbc804b732189db3f
I20240501 20:11:56.680858 26233 tablet_copy_source_session.cc:205] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:56.682067 28316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f145d7f939459fbc804b732189db3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:56.685647 26212 consensus_queue.cc:237] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:56.687341 28316 tablet_copy_client.cc:783] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:56.687649 28316 tablet_copy_client.cc:647] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:56.687934 26077 raft_consensus.cc:1261] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:56.688516 26344 raft_consensus.cc:1261] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:56.689021 28270 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:56.689146 28316 tablet_copy_client.cc:523] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:56.689553 28270 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:56.691787 28316 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap starting.
I20240501 20:11:56.698963 28270 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:56.699611 26344 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:56.703248 25930 catalog_manager.cc:5484] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:56.709167 26212 consensus_queue.cc:237] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
W20240501 20:11:56.712210 26147 consensus_peers.cc:485] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 -> Peer f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545): Couldn't send request to peer f79d4befcb2c4d6cb0e1b79379164c25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3d4fc9946d44e548a0bac91834b8afd. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:56.713753 26344 raft_consensus.cc:1261] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:56.714244 28327 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:56.721524 28270 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:56.724233 25920 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e3d4fc9946d44e548a0bac91834b8afd with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:11:56.724903 26344 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:56.725651 25924 catalog_manager.cc:5484] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 4 to 5, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:56.732105 26076 raft_consensus.cc:1261] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:56.732996 28331 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:56.734174 26076 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:56.737615 26076 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:56.743542 26212 consensus_queue.cc:237] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:56.746397 27151 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:56.746567 28316 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:56.746964 28316 tablet_bootstrap.cc:492] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap complete.
I20240501 20:11:56.747048 26076 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:56.746959 28334 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:56.747272 28316 ts_tablet_manager.cc:1367] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: Time spent bootstrapping tablet: real 0.056s	user 0.035s	sys 0.011s
I20240501 20:11:56.747542 28331 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:56.748070 28316 raft_consensus.cc:348] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:56.748342 28316 raft_consensus.cc:729] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f79d4befcb2c4d6cb0e1b79379164c25, State: Initialized, Role: LEARNER
I20240501 20:11:56.748553 28316 consensus_queue.cc:260] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:56.749240 28316 ts_tablet_manager.cc:1398] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 20:11:56.749855 26233 tablet_copy_service.cc:342] P 41abd2deb0d7412bbb66b1aa9851d888: Request end of tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-97f145d7f939459fbc804b732189db3f received from {username='slave'} at 127.8.68.197:38227
I20240501 20:11:56.749975 26233 tablet_copy_service.cc:433] P 41abd2deb0d7412bbb66b1aa9851d888: ending tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-97f145d7f939459fbc804b732189db3f on tablet 97f145d7f939459fbc804b732189db3f with peer f79d4befcb2c4d6cb0e1b79379164c25
I20240501 20:11:56.750185 27151 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.749670 28334 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.755120 25924 catalog_manager.cc:5484] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.758484 26076 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:56.760849 26212 consensus_queue.cc:237] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:56.763146 27151 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:11:56.763254 26147 consensus_peers.cc:485] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 -> Peer f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197:46545): Couldn't send request to peer f79d4befcb2c4d6cb0e1b79379164c25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd17acc8002948999a5c607ccb9bb189. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:56.763687 26077 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:56.763674 28331 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:56.764211 28331 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:56.766175 28334 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:56.766651 27151 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:56.769428 25920 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cd17acc8002948999a5c607ccb9bb189 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:11:56.771049 25930 catalog_manager.cc:5484] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:56.772295 26076 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } } }
I20240501 20:11:57.064792 27018 raft_consensus.cc:1203] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:57.230556 28316 ts_tablet_manager.cc:902] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: Initiating tablet copy from peer 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:57.231756 28316 tablet_copy_client.cc:308] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:35909
I20240501 20:11:57.232383 26233 tablet_copy_service.cc:140] P 41abd2deb0d7412bbb66b1aa9851d888: Received BeginTabletCopySession request for tablet cd17acc8002948999a5c607ccb9bb189 from peer f79d4befcb2c4d6cb0e1b79379164c25 ({username='slave'} at 127.8.68.197:38227)
I20240501 20:11:57.232540 26233 tablet_copy_service.cc:161] P 41abd2deb0d7412bbb66b1aa9851d888: Beginning new tablet copy session on tablet cd17acc8002948999a5c607ccb9bb189 from peer f79d4befcb2c4d6cb0e1b79379164c25 at {username='slave'} at 127.8.68.197:38227: session id = f79d4befcb2c4d6cb0e1b79379164c25-cd17acc8002948999a5c607ccb9bb189
I20240501 20:11:57.234727 26233 tablet_copy_source_session.cc:205] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:57.235872 28316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd17acc8002948999a5c607ccb9bb189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:57.240571 28385 ts_tablet_manager.cc:902] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: Initiating tablet copy from peer 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:57.240653 28316 tablet_copy_client.cc:783] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:57.240877 28316 tablet_copy_client.cc:647] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:57.242421 28385 tablet_copy_client.cc:308] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:35909
I20240501 20:11:57.242570 28316 tablet_copy_client.cc:523] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:57.244485 26233 tablet_copy_service.cc:140] P 41abd2deb0d7412bbb66b1aa9851d888: Received BeginTabletCopySession request for tablet e3d4fc9946d44e548a0bac91834b8afd from peer f79d4befcb2c4d6cb0e1b79379164c25 ({username='slave'} at 127.8.68.197:38227)
I20240501 20:11:57.244621 26233 tablet_copy_service.cc:161] P 41abd2deb0d7412bbb66b1aa9851d888: Beginning new tablet copy session on tablet e3d4fc9946d44e548a0bac91834b8afd from peer f79d4befcb2c4d6cb0e1b79379164c25 at {username='slave'} at 127.8.68.197:38227: session id = f79d4befcb2c4d6cb0e1b79379164c25-e3d4fc9946d44e548a0bac91834b8afd
I20240501 20:11:57.245411 28316 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap starting.
I20240501 20:11:57.246943 26233 tablet_copy_source_session.cc:205] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:57.248469 28385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d4fc9946d44e548a0bac91834b8afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:57.255506 28385 tablet_copy_client.cc:783] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:57.255730 28385 tablet_copy_client.cc:647] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:57.257169 28385 tablet_copy_client.cc:523] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:57.260550 28385 tablet_bootstrap.cc:492] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap starting.
I20240501 20:11:57.297200 28316 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:57.297638 28316 tablet_bootstrap.cc:492] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap complete.
I20240501 20:11:57.297897 28316 ts_tablet_manager.cc:1367] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: Time spent bootstrapping tablet: real 0.053s	user 0.047s	sys 0.004s
I20240501 20:11:57.298727 28316 raft_consensus.cc:348] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:57.298985 28316 raft_consensus.cc:729] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f79d4befcb2c4d6cb0e1b79379164c25, State: Initialized, Role: LEARNER
I20240501 20:11:57.299212 28316 consensus_queue.cc:260] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:57.299782 28316 ts_tablet_manager.cc:1398] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240501 20:11:57.300521 26233 tablet_copy_service.cc:342] P 41abd2deb0d7412bbb66b1aa9851d888: Request end of tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-cd17acc8002948999a5c607ccb9bb189 received from {username='slave'} at 127.8.68.197:38227
I20240501 20:11:57.300666 26233 tablet_copy_service.cc:433] P 41abd2deb0d7412bbb66b1aa9851d888: ending tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-cd17acc8002948999a5c607ccb9bb189 on tablet cd17acc8002948999a5c607ccb9bb189 with peer f79d4befcb2c4d6cb0e1b79379164c25
I20240501 20:11:57.306922 28385 tablet_bootstrap.cc:492] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:57.307212 28385 tablet_bootstrap.cc:492] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: Bootstrap complete.
I20240501 20:11:57.307394 28385 ts_tablet_manager.cc:1367] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: Time spent bootstrapping tablet: real 0.047s	user 0.040s	sys 0.008s
I20240501 20:11:57.307924 28385 raft_consensus.cc:348] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:57.308100 28385 raft_consensus.cc:729] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f79d4befcb2c4d6cb0e1b79379164c25, State: Initialized, Role: LEARNER
I20240501 20:11:57.308236 28385 consensus_queue.cc:260] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: NON_VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: true } }
I20240501 20:11:57.308671 28385 ts_tablet_manager.cc:1398] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 20:11:57.309320 26233 tablet_copy_service.cc:342] P 41abd2deb0d7412bbb66b1aa9851d888: Request end of tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-e3d4fc9946d44e548a0bac91834b8afd received from {username='slave'} at 127.8.68.197:38227
I20240501 20:11:57.309465 26233 tablet_copy_service.cc:433] P 41abd2deb0d7412bbb66b1aa9851d888: ending tablet copy session f79d4befcb2c4d6cb0e1b79379164c25-e3d4fc9946d44e548a0bac91834b8afd on tablet e3d4fc9946d44e548a0bac91834b8afd with peer f79d4befcb2c4d6cb0e1b79379164c25
I20240501 20:11:57.642009 28270 raft_consensus.cc:1050] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888: attempting to promote NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 to VOTER
I20240501 20:11:57.642680 28270 consensus_queue.cc:237] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:57.644830 27151 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:57.644861 26076 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:57.645511 27018 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:57.645455 28413 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:57.645936 28270 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:57.646225 28334 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:57.649289 28270 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:57.650015 27151 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:57.650090 26076 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:57.656333 27018 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:57.656930 25924 catalog_manager.cc:5484] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 reported cstate change: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:57.663718 26212 consensus_queue.cc:237] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:57.665119 27018 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:57.665421 27151 raft_consensus.cc:1261] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:57.665670 28334 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:57.666465 28334 consensus_queue.cc:1035] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:57.667765 28270 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:57.668265 27018 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:57.669184 27151 raft_consensus.cc:2932] T 97f145d7f939459fbc804b732189db3f P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:57.671541 25920 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97f145d7f939459fbc804b732189db3f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:57.671967 25926 catalog_manager.cc:5484] T 97f145d7f939459fbc804b732189db3f P f79d4befcb2c4d6cb0e1b79379164c25 reported cstate change: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:57.675161 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f145d7f939459fbc804b732189db3f with delete_type TABLET_DATA_TOMBSTONED (TS e9a0d830efeb4d43b13221b36d8e77f0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57318
I20240501 20:11:57.675848 28422 tablet_replica.cc:330] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0: stopping tablet replica
I20240501 20:11:57.676035 28422 raft_consensus.cc:2229] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:57.676391 28422 raft_consensus.cc:2258] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:57.677121 28422 ts_tablet_manager.cc:1875] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:57.681250 28422 ts_tablet_manager.cc:1888] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:11:57.681367 28422 log.cc:1196] T 97f145d7f939459fbc804b732189db3f P e9a0d830efeb4d43b13221b36d8e77f0: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/97f145d7f939459fbc804b732189db3f
I20240501 20:11:57.682068 25920 catalog_manager.cc:4830] TS e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067): tablet 97f145d7f939459fbc804b732189db3f (table rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]) successfully deleted
I20240501 20:11:57.722545 27018 raft_consensus.cc:1203] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 20:11:57.743325 27018 raft_consensus.cc:1203] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 20:11:57.913278 26212 consensus_queue.cc:237] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } }
I20240501 20:11:57.915050 26344 raft_consensus.cc:1261] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:57.915100 26076 raft_consensus.cc:1261] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:57.915593 28334 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:57.915853 28270 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:57.918161 28334 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:57.918804 26344 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:57.919144 26076 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } }
I20240501 20:11:57.922430 25932 catalog_manager.cc:5484] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:57.927312 26212 consensus_queue.cc:237] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:57.929256 26344 raft_consensus.cc:1261] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:57.929332 26076 raft_consensus.cc:1261] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:11:57.929366 26146 consensus_peers.cc:485] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 -> Peer 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196:34631): Couldn't send request to peer 0afaccbf1fb84dd7b530d6632c4cd948. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8625da0d7164bf0b18052e5be5013df. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:57.929874 28334 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:57.930796 28334 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:57.932512 28334 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:57.934867 26344 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:57.935755 26076 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } } }
I20240501 20:11:57.936692 25920 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b8625da0d7164bf0b18052e5be5013df with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:11:57.938697 25932 catalog_manager.cc:5484] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 4 to 5, NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) added. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:58.218643 28334 raft_consensus.cc:1050] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888: attempting to promote NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 to VOTER
I20240501 20:11:58.219291 28334 consensus_queue.cc:237] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:58.221217 27018 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:58.221506 27151 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:58.221580 26076 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:58.221711 28334 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:58.222071 28413 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:58.222486 28477 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:58.225311 28334 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.225857 27018 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.226769 26076 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.229691 27151 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.231339 25932 catalog_manager.cc:5484] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 reported cstate change: config changed from index 8 to 9, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.236265 26212 consensus_queue.cc:237] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:58.237879 27018 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:58.238353 28477 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:58.238667 27151 raft_consensus.cc:1261] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:11:58.239122 28477 consensus_queue.cc:1035] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:58.240170 28334 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.240703 27018 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.241602 27151 raft_consensus.cc:2932] T cd17acc8002948999a5c607ccb9bb189 P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.244346 25920 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd17acc8002948999a5c607ccb9bb189 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:58.244553 25924 catalog_manager.cc:5484] T cd17acc8002948999a5c607ccb9bb189 P f79d4befcb2c4d6cb0e1b79379164c25 reported cstate change: config changed from index 9 to 10, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.248021 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet cd17acc8002948999a5c607ccb9bb189 with delete_type TABLET_DATA_TOMBSTONED (TS e9a0d830efeb4d43b13221b36d8e77f0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57318
I20240501 20:11:58.253909 28482 tablet_replica.cc:330] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0: stopping tablet replica
I20240501 20:11:58.254091 28482 raft_consensus.cc:2229] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:58.254371 28482 raft_consensus.cc:2258] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:58.255138 28482 ts_tablet_manager.cc:1875] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:58.259275 28482 ts_tablet_manager.cc:1888] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:11:58.259366 28482 log.cc:1196] T cd17acc8002948999a5c607ccb9bb189 P e9a0d830efeb4d43b13221b36d8e77f0: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/cd17acc8002948999a5c607ccb9bb189
I20240501 20:11:58.259976 25920 catalog_manager.cc:4830] TS e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067): tablet cd17acc8002948999a5c607ccb9bb189 (table rebalance_test_table_0 [id=f82789acf5694d8a84a0613aa6436520]) successfully deleted
I20240501 20:11:58.307401 28334 raft_consensus.cc:1050] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888: attempting to promote NON_VOTER f79d4befcb2c4d6cb0e1b79379164c25 to VOTER
I20240501 20:11:58.308029 28334 consensus_queue.cc:237] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:58.310062 26076 raft_consensus.cc:1261] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:58.310060 27018 raft_consensus.cc:1261] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 LEARNER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:58.310060 26344 raft_consensus.cc:1261] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:58.310652 28413 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:58.310978 28270 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:58.311265 28334 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:58.313855 28413 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.315038 27018 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.316743 26076 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.318836 25926 catalog_manager.cc:5484] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25 reported cstate change: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.321416 26344 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f79d4befcb2c4d6cb0e1b79379164c25 (127.8.68.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.323936 26212 consensus_queue.cc:237] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } }
I20240501 20:11:58.325867 27018 raft_consensus.cc:1261] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:58.326056 26344 raft_consensus.cc:1261] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:58.326370 28477 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:58.326756 28334 consensus_queue.cc:1035] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:58.328614 28477 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.329241 26344 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.329236 27018 raft_consensus.cc:2932] T e3d4fc9946d44e548a0bac91834b8afd P f79d4befcb2c4d6cb0e1b79379164c25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } } }
I20240501 20:11:58.331221 25920 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e3d4fc9946d44e548a0bac91834b8afd with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:58.332257 25930 catalog_manager.cc:5484] T e3d4fc9946d44e548a0bac91834b8afd P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4befcb2c4d6cb0e1b79379164c25" member_type: VOTER last_known_addr { host: "127.8.68.197" port: 46545 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:58.335530 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d4fc9946d44e548a0bac91834b8afd with delete_type TABLET_DATA_TOMBSTONED (TS e9a0d830efeb4d43b13221b36d8e77f0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57318
I20240501 20:11:58.336309 28482 tablet_replica.cc:330] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0: stopping tablet replica
I20240501 20:11:58.336467 28482 raft_consensus.cc:2229] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:58.336649 28482 raft_consensus.cc:2258] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:58.337417 28482 ts_tablet_manager.cc:1875] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:58.341661 28482 ts_tablet_manager.cc:1888] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:11:58.341765 28482 log.cc:1196] T e3d4fc9946d44e548a0bac91834b8afd P e9a0d830efeb4d43b13221b36d8e77f0: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/e3d4fc9946d44e548a0bac91834b8afd
I20240501 20:11:58.342419 25920 catalog_manager.cc:4830] TS e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067): tablet e3d4fc9946d44e548a0bac91834b8afd (table rebalance_test_table_1 [id=4a25e47283bf411c98b921c779bbcad8]) successfully deleted
I20240501 20:11:58.456037 28502 ts_tablet_manager.cc:902] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: Initiating tablet copy from peer 41abd2deb0d7412bbb66b1aa9851d888 (127.8.68.194:35909)
I20240501 20:11:58.457186 28502 tablet_copy_client.cc:308] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Beginning tablet copy session from remote peer at address 127.8.68.194:35909
I20240501 20:11:58.457805 26233 tablet_copy_service.cc:140] P 41abd2deb0d7412bbb66b1aa9851d888: Received BeginTabletCopySession request for tablet b8625da0d7164bf0b18052e5be5013df from peer 0afaccbf1fb84dd7b530d6632c4cd948 ({username='slave'} at 127.8.68.196:55851)
I20240501 20:11:58.457949 26233 tablet_copy_service.cc:161] P 41abd2deb0d7412bbb66b1aa9851d888: Beginning new tablet copy session on tablet b8625da0d7164bf0b18052e5be5013df from peer 0afaccbf1fb84dd7b530d6632c4cd948 at {username='slave'} at 127.8.68.196:55851: session id = 0afaccbf1fb84dd7b530d6632c4cd948-b8625da0d7164bf0b18052e5be5013df
I20240501 20:11:58.459877 26233 tablet_copy_source_session.cc:205] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:58.460973 28502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8625da0d7164bf0b18052e5be5013df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:58.465189 28502 tablet_copy_client.cc:783] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:58.465405 28502 tablet_copy_client.cc:647] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:58.466790 28502 tablet_copy_client.cc:523] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:58.468880 28502 tablet_bootstrap.cc:492] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap starting.
I20240501 20:11:58.496680 28502 tablet_bootstrap.cc:492] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:58.496971 28502 tablet_bootstrap.cc:492] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: Bootstrap complete.
I20240501 20:11:58.497172 28502 ts_tablet_manager.cc:1367] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent bootstrapping tablet: real 0.028s	user 0.023s	sys 0.004s
I20240501 20:11:58.497730 28502 raft_consensus.cc:348] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:58.497912 28502 raft_consensus.cc:729] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0afaccbf1fb84dd7b530d6632c4cd948, State: Initialized, Role: LEARNER
I20240501 20:11:58.498055 28502 consensus_queue.cc:260] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: NON_VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: true } }
I20240501 20:11:58.498500 28502 ts_tablet_manager.cc:1398] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240501 20:11:58.499081 26233 tablet_copy_service.cc:342] P 41abd2deb0d7412bbb66b1aa9851d888: Request end of tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-b8625da0d7164bf0b18052e5be5013df received from {username='slave'} at 127.8.68.196:55851
I20240501 20:11:58.499213 26233 tablet_copy_service.cc:433] P 41abd2deb0d7412bbb66b1aa9851d888: ending tablet copy session 0afaccbf1fb84dd7b530d6632c4cd948-b8625da0d7164bf0b18052e5be5013df on tablet b8625da0d7164bf0b18052e5be5013df with peer 0afaccbf1fb84dd7b530d6632c4cd948
I20240501 20:11:59.020534 27151 raft_consensus.cc:1203] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 20:11:59.439633 28518 raft_consensus.cc:1050] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888: attempting to promote NON_VOTER 0afaccbf1fb84dd7b530d6632c4cd948 to VOTER
I20240501 20:11:59.440271 28518 consensus_queue.cc:237] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:59.442533 27151 raft_consensus.cc:1261] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 LEARNER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:59.442546 26344 raft_consensus.cc:1261] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:59.442687 26076 raft_consensus.cc:1261] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:59.443109 28518 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:59.443567 28532 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:59.443897 28519 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:59.446761 28519 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:59.450057 26076 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:59.453410 27151 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:59.454121 25926 catalog_manager.cc:5484] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 5 to 6, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:59.457039 26344 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0afaccbf1fb84dd7b530d6632c4cd948 (127.8.68.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "e9a0d830efeb4d43b13221b36d8e77f0" member_type: VOTER last_known_addr { host: "127.8.68.193" port: 41067 } attrs { replace: true } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:59.469554 26212 consensus_queue.cc:237] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } }
I20240501 20:11:59.471100 26344 raft_consensus.cc:1261] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:59.471217 27151 raft_consensus.cc:1261] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Refusing update from remote peer 41abd2deb0d7412bbb66b1aa9851d888: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:59.471537 28518 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:59.471804 28519 consensus_queue.cc:1035] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:59.474117 28518 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:59.475003 26344 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 1c992b98bb0d4f1b92d058c882fb68b9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:59.477150 27151 raft_consensus.cc:2932] T b8625da0d7164bf0b18052e5be5013df P 0afaccbf1fb84dd7b530d6632c4cd948 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } } }
I20240501 20:11:59.477651 25920 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8625da0d7164bf0b18052e5be5013df with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:59.478442 25932 catalog_manager.cc:5484] T b8625da0d7164bf0b18052e5be5013df P 41abd2deb0d7412bbb66b1aa9851d888 reported cstate change: config changed from index 6 to 7, VOTER e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193) evicted. New cstate: current_term: 1 leader_uuid: "41abd2deb0d7412bbb66b1aa9851d888" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "41abd2deb0d7412bbb66b1aa9851d888" member_type: VOTER last_known_addr { host: "127.8.68.194" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c992b98bb0d4f1b92d058c882fb68b9" member_type: VOTER last_known_addr { host: "127.8.68.195" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afaccbf1fb84dd7b530d6632c4cd948" member_type: VOTER last_known_addr { host: "127.8.68.196" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:59.482262 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet b8625da0d7164bf0b18052e5be5013df with delete_type TABLET_DATA_TOMBSTONED (TS e9a0d830efeb4d43b13221b36d8e77f0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57318
I20240501 20:11:59.483000 28546 tablet_replica.cc:330] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0: stopping tablet replica
I20240501 20:11:59.483211 28546 raft_consensus.cc:2229] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:59.483474 28546 raft_consensus.cc:2258] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:59.484972 28546 ts_tablet_manager.cc:1875] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:59.488961 28546 ts_tablet_manager.cc:1888] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:11:59.489070 28546 log.cc:1196] T b8625da0d7164bf0b18052e5be5013df P e9a0d830efeb4d43b13221b36d8e77f0: Deleting WAL directory at /tmp/dist-test-taskH_e3f3/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714594130440066-8467-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8625da0d7164bf0b18052e5be5013df
I20240501 20:11:59.489755 25920 catalog_manager.cc:4830] TS e9a0d830efeb4d43b13221b36d8e77f0 (127.8.68.193:41067): tablet b8625da0d7164bf0b18052e5be5013df (table rebalance_test_table_2 [id=441c6a38db704bce869d0b07388eeba9]) successfully deleted
I20240501 20:11:59.774076 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:59.785526 26056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.795809 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.803447 26999 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:59.815658 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:59.948298 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.957406 26999 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:59.964810 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:59.981765 26056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.983297 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:00.138473 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.139321 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:00.150538 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:00.166517 26056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.167671 26999 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:00.307363 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.313822 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:00.324326 26056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.333921 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:00.347033 26999 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:00.477607 27131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.477591 26325 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:00.496639 26191 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:00.497674 26056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:00.514978 26999 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:00.701073  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 25994
I20240501 20:12:00.709575 26119 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:00.854918  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 25994
I20240501 20:12:00.878446  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 26130
I20240501 20:12:00.884238 26252 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:01.047479  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 26130
I20240501 20:12:01.070834  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 26262
I20240501 20:12:01.076493 26384 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:01.243228  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 26262
W20240501 20:12:01.265872 27086 connection.cc:535] server connection from 127.8.68.195:41139 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:12:01.266330  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 27069
I20240501 20:12:01.272754 27190 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:01.403657  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 27069
I20240501 20:12:01.423127  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 26937
I20240501 20:12:01.429490 27059 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:01.565704  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 26937
I20240501 20:12:01.586138  8467 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu pid 25902
I20240501 20:12:01.587189 25964 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:01.740044  8467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH_e3f3/build/asan/bin/kudu with pid 25902
2024-05-01T20:12:01Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (25204 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (46922 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (191332 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00>

 1 FAILED TEST